Tensorflow2 Object Detection API tutorial: 신호등 객체 검출

[2001207 이단유]

참고 링크

https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/index.html

참고 문헌

http://www.koreascience.or.kr/article/JAKO202219559499578.pdf 

데이터셋 클래스 총 7개: green, greenleft, red, redleft, redright, soright, yellow

훈련 데이터- 클래스별 9개씩 총 63개

테스트 데이터- 클래스별 1개씩 총 7개

# 데이터 수집에 어려움 있음

1. 설치

1) Anaconda Python 3.9 

- 환경 변수에 Anaconda3 추가 확인란 선택

2) TensorFlow

- version 2.5.0

- GPU 지원: CUDA toolkit, cuDNN(Window용), 환경 변수 편집

3) TensorFlow Object Detection API 

- Tensorflow Model Garden 다운로드

새 폴더 생성 후 model_master.zip 다운로드

- Protobuf 설치/컴파일

protoc-3.20.3-win64 설치 => 최신 버전 설치시 오류 발생

- COCO API 설치

- 객체 감지 API 설치 오류

 =>  copy object_detection\\packages\\tf2\\setup.py .로 바꿔서 입력

- 설치 테스트 오류

=> pip install object_detection 입력 후 다시 실행

=> protoc object_detection/protos/*.proto --python_out=.

pip install .

입력 후 재실행

=> pip install --upgrade protobuf 실행 후 anaconda3 가상 환경 site-packages/google/protobuf/internal 폴더에서 builder.py 파일을 Tensorflow 폴더로 복사

pip install protobuf==3.19.4 실행 후 Tensorflow 폴더에 있는 builder.py 다시 가상 환경 폴더로 복사 후 재실행

2. Training Custom Object Detector

1) 작업 공간 준비: annotations, exported-models, images, models, pre-trained-models, README 파일을 포함한 workspace 폴더 생성

2) 데이터 세트 준비

- 데이터 세트에 주석 달기

labelImg 설치 후 images 파일 안에 이미지들을 넣고 이미지 라벨링 시키기-> 모든 이미지에 각 하나씩 정보를 담고 있는 xml 파일이 생성됨

클래스별 샘플 이미지

green

greenleft

red

redleft

redright

soright

yellow

- 데이터 세트 분할

train 이미지와 test 이미지를 분할하여 각각의 폴더로 이동

- 레이블 맵 생성

label_map.pbtxt 파일 생성하여 annotations 폴더에 저장-> 각각의 id와 name을 딕셔너리 형태로 파일에 저장

- TensorFlow 레코드 만들기

예제 코드를 실행하여 train과 test 폴더 안에 있던 *.xml 파일을 두 개의 *.record 파일을 생성

3) 교육 작업 구성

- 사전 학습된 모델 다운로드

pre-trained-models 폴더에 SSD ResNet50 V1 FPN 640x640 모델 다운로드

- 교육 파이프라인 구성

pipeline.config 파일에 num_classes, batch_size, fine_tune_checkpoint, fine_tune_checkpoint_type, use_bfloat16, label_map_path, input_path, metrics_set, use_moving_averages 변수 편집하고 저장

4) 모델 훈련

5) TensorBoard를 사용하여 교육 작업 진행률 모니터링

classification_loss, localization_loss, regularization_loss 각각 분류 손실, 지역화 손실, 정규화 손실을 나타낸 그래프

learning_rate: 학습률

=> 모델 훈련에 소요된 시간: 약 3시간

6) 훈련된 모델 내보내기

research/object_detection/exporter_main_v2.py 파일을 training_demo 폴더에서 실행하면 모델 내보내기 완료

3. 예제 실행

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443

import os os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'    # Suppress TensorFlow logging (1) import pathlib import tensorflow as tf import xml.etree.ElementTree as ET   tf.get_logger().setLevel('ERROR')           # Suppress TensorFlow logging (2)   # Enable GPU dynamic memory allocation gpus = tf.config.experimental.list_physical_devices('GPU') for gpu in gpus:     tf.config.experimental.set_memory_growth(gpu, True)   # 이미지 파일 불러오기 def download_images():     base_url = 'D:/ldy/Tensorflow/tl_workspace/training_demo/images/test/'     filenames=[]     for f in os.listdir('D:/ldy/Tensorflow/tl_workspace/training_demo/images/test'):         filenames.append(f)     image_paths = []     for filename in filenames:         image_path = pathlib.Path(base_url+filename)         image_paths.append(str(image_path))     return image_paths   IMAGE_PATHS = download_images()   # xml 파일 불러오기 def download_xmls():     base_url = 'D:/ldy/Tensorflow/tl_workspace/training_demo/images/testXml/'     filenames=[]     for f in os.listdir('D:/ldy/Tensorflow/tl_workspace/training_demo/images/testXml'):         filenames.append(f)     xml_paths = []     for filename in filenames:         xml_path = pathlib.Path(base_url+filename)         xml_paths.append(str(xml_path))     return xml_paths   XML_PATHS=download_xmls()   # Download labels file def download_labels(filename):     base_url = 'D:/ldy/Tensorflow/tl_workspace/training_demo/annotations/'     label_dir = pathlib.Path(base_url+LABEL_FILENAME)     return str(label_dir)   LABEL_FILENAME = 'label_map.pbtxt' PATH_TO_LABELS = download_labels(LABEL_FILENAME)   # Load the model # ~~~~~~~~~~~~~~ # Next we load the downloaded model import time from object_detection.utils import label_map_util from object_detection.utils import visualization_utils as viz_utils   # saved_model 경로 PATH_TO_SAVED_MODEL = "D:/ldy/Tensorflow/tl_workspace/training_demo/exported-models/my_model/ saved_model"   print('Loading model...', end='') start_time = time.time()   # Load saved model and build the detection function detect_fn = tf.saved_model.load(PATH_TO_SAVED_MODEL)   end_time = time.time() elapsed_time = end_time - start_time print('Done! Took {} seconds'.format(elapsed_time))   # Load label map data (for plotting) category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS,                                                                     use_display_name=True)   # Putting everything together import numpy as np from PIL import Image import matplotlib.pyplot as plt import warnings import cv2 import math warnings.filterwarnings('ignore')   # Suppress Matplotlib warnings   def load_image_into_numpy_array(path):     """Load an image from file into a numpy array.     Puts image into numpy array to feed into tensorflow graph.     Note that by convention we put it into a numpy array with shape     (height, width, channels), where channels=3 for RGB.     Args:       path: the file path to the image     Returns:       uint8 numpy array with shape (img_height, img_width, 3)     """     return np.array(Image.open(path))   allCount=0 ans=0 for image_path in IMAGE_PATHS:     # xml 파일 파싱     xml_path=image_path.replace('.jpg','.xml')     xml_path=xml_path.replace('test','testXml')     tree=ET.parse(xml_path)     root = tree.getroot()       # xml 파일에서 정답 좌표값 검색 후 변수에 저장     real_pt=[]     for object in root.findall("object"):         name = object.find("name")         real_name = name.text         bndbox=object.find("bndbox")         xmin=bndbox.find("xmin")         ymin=bndbox.find("ymin")         real_xmin=int(xmin.text) # 바운딩박스 좌표값 받아오기         real_ymin=int(ymin.text) # 바운딩박스 좌표값 받아오기         print(real_xmin, real_ymin)         real_pt.append([real_xmin,real_ymin]) # [xmin, ymin] 리스트로 저장         print(real_pt)       print('Running inference for {}... '.format(image_path), end='\n')     image_np = load_image_into_numpy_array(image_path)       # The input needs to be a tensor, convert it using `tf.convert_to_tensor`.     input_tensor = tf.convert_to_tensor(image_np)     # The model expects a batch of images, so add an axis with `tf.newaxis`.     input_tensor = input_tensor[tf.newaxis, ...]     # input_tensor = np.expand_dims(image_np, 0)     detections = detect_fn(input_tensor)       # 예측     num_detections = int(detections.pop('num_detections'))     detections = {key: value[0, :num_detections].numpy()                     for key, value in detections.items()}     detections['num_detections'] = num_detections       # detection_classes should be ints.     detections['detection_classes'] = detections['detection_classes'].astype(np.int64)     image_np_with_detections = image_np.copy()       # 이미지 불러오기     image=cv2.imread(image_path)     h,w,c=image.shape # 이미지에서 height, width, channel 정보 가져오기       # pbtxt 파일 읽기     infile=open("d:\\ldy\\Tensorflow\\tl_workspace\\training_demo\\annotations\\label_map.pbtxt", "r")     category = {}     category_name = []     category_id=[]     lines=infile.readlines()       for line in lines:         line=line.replace(' ','')         line=line.lstrip()         line=line.rstrip("\n")         line=line.split(":")         if line[0]=="item{" or line[0]== "}":             continue         # index 번호 category_id 리스트에 추가         elif line[0]=="id":             category_id.append(line[1])         # category_name 리스트에 레이블명 추가         elif line[0]=="name":             line[1]=line[1].replace('\'','')             category_name.append(line[1])         else:             continue          # category 딕셔너리에 키:값 = 인덱스:레이블명 형태로 저장     for i in range(len(category_id)):         category[category_id[i]]=category_name[i]           for n in range(len(real_pt)):         allCount+=1         scoredict={}         for i in range(len(detections['detection_boxes'])):              pt=detections['detection_boxes'][i]              if detections['detection_scores'][i]>0.3:                 # 예측 바운딩 박스 상대 좌표를 절대 좌표로 변환                 (xmin, ymin, xmax, ymax) = (int(pt[1] * w), int(pt[0] * h), int(pt[3] * w),  int(pt[2] * h))                   a = abs(real_pt[n][0]-xmin)                 b = abs(real_pt[n][1]-ymin)                    c = math.sqrt((a * a) + (b * b)) # 두 점 사이 거리 계산                   # 정답 바운딩 박스와 예측 바운딩 박스의 거리가 일정 미만일때 예측 결과 출력                 if c<=5.0:                     cv2.rectangle(image,(xmin,ymin),(xmax,ymax),(0,0,255),2)                     for k in range(len(category_id)):                         if category_id[k]==str(detections['detection_classes'][i]):                             if category[category_id[k]]==real_name:                                 ans+=1                             scoredict[category[category_id[k]]]=str(detections['detection_scores'] [i])                             print(scoredict)                             print(xmin,ymin,xmax,ymax)                             if len(scoredict)==0:                                 continue                             print("detection_class:",category[category_id[k]]+", real_class:", real_name+", detection_score:",                                   str(int(detections['detection_scores'][i]*100))+"%")                             m=max(scoredict,key=scoredict.get)                             # print(m)                             cv2.putText(image,m+", "+str(int(float(scoredict[m])*100))+"%",                                         (xmin+5,ymax-5),cv2.FONT_HERSHEY_PLAIN,1.0,(128,128,128))                 else:                     continue       cv2.imshow("image",image)     cv2.waitKey()   # 정확도 계산 print(f"바운딩박스 총 개수: {allCount}, 맞춘 개수: {ans}, 정확도: {ans/allCount*100}%")   # sphinx_gallery_thumbnail_number = 2 import os os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'    # Suppress TensorFlow logging (1) import pathlib import tensorflow as tf import xml.etree.ElementTree as ET   tf.get_logger().setLevel('ERROR')           # Suppress TensorFlow logging (2)   # Enable GPU dynamic memory allocation gpus = tf.config.experimental.list_physical_devices('GPU') for gpu in gpus:     tf.config.experimental.set_memory_growth(gpu, True)   # 이미지 파일 불러오기 def download_images():     base_url = 'D:/ldy/Tensorflow/tl_workspace/training_demo/images/test/'     filenames=[]     for f in os.listdir('D:/ldy/Tensorflow/tl_workspace/training_demo/images/test'):         filenames.append(f)     image_paths = []     for filename in filenames:         image_path = pathlib.Path(base_url+filename)         image_paths.append(str(image_path))     return image_paths   IMAGE_PATHS = download_images()   # xml 파일 불러오기 def download_xmls():     base_url = 'D:/ldy/Tensorflow/tl_workspace/training_demo/images/testXml/'     filenames=[]     for f in os.listdir('D:/ldy/Tensorflow/tl_workspace/training_demo/images/testXml'):         filenames.append(f)     xml_paths = []     for filename in filenames:         xml_path = pathlib.Path(base_url+filename)         xml_paths.append(str(xml_path))     return xml_paths   XML_PATHS=download_xmls()   # Download labels file def download_labels(filename):     base_url = 'D:/ldy/Tensorflow/tl_workspace/training_demo/annotations/'     label_dir = pathlib.Path(base_url+LABEL_FILENAME)     return str(label_dir)   LABEL_FILENAME = 'label_map.pbtxt' PATH_TO_LABELS = download_labels(LABEL_FILENAME)   # Load the model # ~~~~~~~~~~~~~~ # Next we load the downloaded model import time from object_detection.utils import label_map_util from object_detection.utils import visualization_utils as viz_utils   # saved_model 경로 PATH_TO_SAVED_MODEL = "D:/ldy/Tensorflow/tl_workspace/training_demo/exported-models/my_model/ saved_model"   print('Loading model...', end='') start_time = time.time()   # Load saved model and build the detection function detect_fn = tf.saved_model.load(PATH_TO_SAVED_MODEL)   end_time = time.time() elapsed_time = end_time - start_time print('Done! Took {} seconds'.format(elapsed_time))   # Load label map data (for plotting) category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS,                                                                     use_display_name=True)   # Putting everything together import numpy as np from PIL import Image import matplotlib.pyplot as plt import warnings import cv2 import math warnings.filterwarnings('ignore')   # Suppress Matplotlib warnings   def load_image_into_numpy_array(path):     """Load an image from file into a numpy array.     Puts image into numpy array to feed into tensorflow graph.     Note that by convention we put it into a numpy array with shape     (height, width, channels), where channels=3 for RGB.     Args:       path: the file path to the image     Returns:       uint8 numpy array with shape (img_height, img_width, 3)     """     return np.array(Image.open(path))   allCount=0 ans=0 for image_path in IMAGE_PATHS:     # xml 파일 파싱     xml_path=image_path.replace('.jpg','.xml')     xml_path=xml_path.replace('test','testXml')     tree=ET.parse(xml_path)     root = tree.getroot()       # xml 파일에서 정답 좌표값 검색 후 변수에 저장     real_pt=[]     for object in root.findall("object"):         name = object.find("name")         real_name = name.text         bndbox=object.find("bndbox")         xmin=bndbox.find("xmin")         ymin=bndbox.find("ymin")         real_xmin=int(xmin.text) # 바운딩박스 좌표값 받아오기         real_ymin=int(ymin.text) # 바운딩박스 좌표값 받아오기         print(real_xmin, real_ymin)         real_pt.append([real_xmin,real_ymin]) # [xmin, ymin] 리스트로 저장         print(real_pt)         # print(type(real_pt))       print('Running inference for {}... '.format(image_path), end='\n')     image_np = load_image_into_numpy_array(image_path)       # The input needs to be a tensor, convert it using `tf.convert_to_tensor`.     input_tensor = tf.convert_to_tensor(image_np)     # The model expects a batch of images, so add an axis with `tf.newaxis`.     input_tensor = input_tensor[tf.newaxis, ...]     # input_tensor = np.expand_dims(image_np, 0)     detections = detect_fn(input_tensor)       # All outputs are batches tensors.     # Convert to numpy arrays, and take index [0] to remove the batch dimension.     # We're only interested in the first num_detections.     num_detections = int(detections.pop('num_detections'))     detections = {key: value[0, :num_detections].numpy()                     for key, value in detections.items()}     detections['num_detections'] = num_detections       # detection_classes should be ints.     detections['detection_classes'] = detections['detection_classes'].astype(np.int64)     image_np_with_detections = image_np.copy()       # 이미지 불러오기     image=cv2.imread(image_path)     h,w,c=image.shape # 이미지에서 height, width, channel 정보 가져오기       # pbtxt 파일 읽기     infile=open("d:\\ldy\\Tensorflow\\tl_workspace\\training_demo\\annotations\\label_map.pbtxt", "r")     category = {}     category_name = []     category_id=[]     lines=infile.readlines()       for line in lines:         line=line.replace(' ','')         line=line.lstrip()         line=line.rstrip("\n")         line=line.split(":")         if line[0]=="item{" or line[0]== "}":             continue         # index 번호 category_id 리스트에 추가         elif line[0]=="id":             category_id.append(line[1])         # category_name 리스트에 레이블명 추가         elif line[0]=="name":             line[1]=line[1].replace('\'','')             category_name.append(line[1])         else:             continue          # category 딕셔너리에 키:값 = 인덱스:레이블명 형태로 저장     for i in range(len(category_id)):         category[category_id[i]]=category_name[i]           for n in range(len(real_pt)):         allCount+=1         scoredict={}         for i in range(len(detections['detection_boxes'])):              pt=detections['detection_boxes'][i]              if detections['detection_scores'][i]>0.3:                 # 상대 좌표 절대 좌표로 변환                 (xmin, ymin, xmax, ymax) = (int(pt[1] * w), int(pt[0] * h), int(pt[3] * w),  int(pt[2] * h))                   a = abs(real_pt[n][0]-xmin)                 b = abs(real_pt[n][1]-ymin)                    c = math.sqrt((a * a) + (b * b)) # 두 점 사이 거리 계산                   # 정답 바운딩 박스와 예측 바운딩 박스의 거리가 일정 미만일때 예측 결과 출력                 if c<=5.0:                     cv2.rectangle(image,(xmin,ymin),(xmax,ymax),(0,0,255),2)                     for k in range(len(category_id)):                         if category_id[k]==str(detections['detection_classes'][i]):                             if category[category_id[k]]==real_name:                                 ans+=1                             scoredict[category[category_id[k]]]=str(detections['detection_scores'] [i])                             print(scoredict)                             print(xmin,ymin,xmax,ymax)                             if len(scoredict)==0:                                 continue                             print("detection_class:",category[category_id[k]]+", real_class:", real_name+", detection_score:",                                   str(int(detections['detection_scores'][i]*100))+"%")                             m=max(scoredict,key=scoredict.get)                             # print(m)                             cv2.putText(image,m+", "+str(int(float(scoredict[m])*100))+"%",                                         (xmin+5,ymax-5),cv2.FONT_HERSHEY_PLAIN,1.0,(128,128,128))                 else:                     continue       cv2.imshow("image",image)     cv2.waitKey()   # 정확도 계산 print(f"바운딩박스 총 개수: {allCount}, 맞춘 개수: {ans}, 정확도: {ans/allCount*100}%")   # sphinx_gallery_thumbnail_number = 2 Colored by Color Scripter

cs

첫번째 실행 결과 정확도: 28%

=> pipeline.config 파일에서 data_augmentation_options 파라미터 이용하여 데이터 증식

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

data_augmentation_options {     random_vertical_flip{     }     random_adjust_brightness{     }     random_adjust_contrast{     }     random_adjust_saturation{     }     random_patch_gaussian{     }     ssd_random_crop {     }     random_pixel_value_scale {       minval: 0.6     }     random_crop_image {       min_object_covered: 0.0       min_aspect_ratio: 0.75       max_aspect_ratio: 3.0       min_area: 0.75       max_area: 1.0       overlap_thresh: 0.0     }   }

cs

모델 훈련 다시 시킨 후 정확도 재출력

두번째 실행 결과 정확도: 50%

세번째 실행 결과 정확도: 57%

=> data augmentations(데이터 증식)을 통해 정확도 향상시킴

# 신호등의 클래스 분류

논문에서는 신호등 클래스를 상기와 같이 분류함

=> 정지 신호와 좌회전(우회전) 신호가 같이 켜져있는 것과 좌회전(우회전) 신호만 켜져있는 것은 같은 클래스로 분류하여 클래스 5개로 다시 훈련 시켜봐야 할 것 같음

[Sungryul Lee]

1.다음부터는 함수로 만들어서 재활용할수 있게 코딩할것
2.클래스별로 샘플이미지 추가할것
3.정확도를 높일려면 신호등의 클래스분류를 고민해봐야 할것 같다.
신호등인식을 남들은 어떻게 하는지 검색해봐라 ,레이블링을 어떻게 하는지, 방향등 처리를 어떻게 하는지