Computer >> 컴퓨터 >  >> 프로그래밍 >> Python

TensorFlow를 사용해 Python에서 IMDB 데이터셋 다운로드 및 탐색하는 방법

TensorFlow란 무엇인가?

TensorFlow는 Google이 제공하는 머신러닝 프레임워크입니다. 오픈소스로 공개되어 있으며, Python과 함께 사용하여 알고리즘 구현, 딥러닝 애플리케이션 개발 등 다양한 작업에 활용할 수 있습니다. 연구 목적과 실제 프로덕션 환경 모두에서 널리 사용되고 있습니다.

TensorFlow는 NumPy와 다차원 배열을 기반으로 동작합니다. 이러한 다차원 배열은 '텐서(tensor)'라고 불립니다. 이 프레임워크는 딥러닝 신경망 작업을 지원하며, 뛰어난 확장성을 갖추고 있고 다양한 인기 데이터셋을 함께 제공합니다. GPU 연산을 활용하고 리소스 관리를 자동화하며, 수많은 머신러닝 라이브러리를 포함하고 있어 문서화와 커뮤니티 지원도 매우 잘 되어 있습니다. TensorFlow를 사용하면 딥러닝 신경망 모델을 실행하고 학습시킬 수 있으며, 데이터셋의 특성을 예측하는 애플리케이션까지 손쉽게 만들 수 있습니다.

TensorFlow 설치하기

'tensorflow' 패키지는 Windows 환경에서 아래 명령어 한 줄로 간단히 설치할 수 있습니다.

pip install tensorflow

텐서(Tensor)란?

텐서는 TensorFlow에서 사용되는 핵심 데이터 구조입니다. 플로우 다이어그램의 엣지(edge)를 연결하는 역할을 하며, 이러한 다이어그램은 '데이터 흐름 그래프(Data Flow Graph)'라고 합니다. 텐서는 본질적으로 다차원 배열 또는 리스트에 해당하며, 세 가지 주요 속성으로 식별할 수 있습니다.

IMDB 데이터셋 소개

'IMDB' 데이터셋은 5만 편이 넘는 영화 리뷰를 담고 있습니다. 이 데이터셋은 일반적으로 자연어 처리(NLP) 관련 작업에 활용됩니다.

이 글의 코드는 Google Colaboratory(Colab)에서 실행됩니다. Google Colab은 브라우저에서 바로 Python 코드를 실행할 수 있게 해주며, 별도의 설정이 필요 없고 GPU에 무료로 접근할 수 있다는 큰 장점이 있습니다. Colaboratory는 Jupyter Notebook을 기반으로 만들어졌습니다.

코드 예제

import matplotlib.pyplot as plt
import os
import re
import shutil
import string
import tensorflow as tf

from tensorflow.keras import layers
from tensorflow.keras import losses
from tensorflow.keras import preprocessing
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization
print("The tensorflow version is ")
print(tf.__version__)
url = "https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz"

dataset = tf.keras.utils.get_file("aclImdb_v1.tar.gz", url,
                                  untar=True, cache_dir='.',
                                  cache_subdir='')
print("The dataset is being downloaded")
dataset_dir = os.path.join(os.path.dirname(dataset), 'aclImdb')
print("The directories in the downloaded folder are ")
os.listdir(dataset_dir)

train_dir = os.path.join(dataset_dir, 'train')
os.listdir(train_dir)
print("The sample of data : ")
sample_file = os.path.join(train_dir, 'pos/1181_9.txt')
with open(sample_file) as f:
  print(f.read())

remove_dir = os.path.join(train_dir, 'unsup')
shutil.rmtree(remove_dir)
batch_size = 32
seed = 42
print("The batch size is")
print(batch_size)

raw_train_ds = tf.keras.preprocessing.text_dataset_from_directory(
    'aclImdb/train',
    batch_size=batch_size,
    validation_split=0.2,
    subset='training',
    seed=seed)

for text_batch, label_batch in raw_train_ds.take(1):
  for i in range(3):
    print("Review", text_batch.numpy()[i])
    print("Label", label_batch.numpy()[i])

print("Label 0 corresponds to", raw_train_ds.class_names[0])
print("Label 1 corresponds to", raw_train_ds.class_names[1])
raw_val_ds = tf.keras.preprocessing.text_dataset_from_directory(
    'aclImdb/train',
    batch_size=batch_size,
    validation_split=0.2,
    subset='validation',
    seed=seed)
raw_test_ds = tf.keras.preprocessing.text_dataset_from_directory(
    'aclImdb/test',
    batch_size=batch_size)

코드 출처 − https://www.tensorflow.org/tutorials/keras/text_classification

실행 결과

The tensorflow version is
2.4.0
The dataset is being downloaded
The directories in the downloaded folder are
The sample of data :
Rachel Griffiths writes and directs this award winning short film. A heartwarming story about coping with grief and cherishing the memory of those we've loved and lost. Although, only 15 minutes long, Griffiths manages to capture so much emotion and truth onto film in the short space of time. Bud Tingwell gives a touching performance as Will, a widower struggling to cope with his wife's death. Will is confronted by the harsh reality of loneliness and helplessness as he proceeds to take care of Ruth's pet cow, Tulip. The film displays the grief and responsibility one feels for those they have loved and lost. Good cinematography, great direction, and superbly acted. It will bring tears to all those who have lost a loved one, and survived.
The batch size is
32
Found 25000 files belonging to 2 classes.
Using 20000 files for training.
Review b'"Pandemonium" is a horror movie spoof that comes off more stupid than funny. Believe me when I tell you, I love comedies. Especially comedy spoofs. "Airplane", "The Naked Gun" trilogy, "Blazing Saddles", "High Anxiety", and "Spaceballs" are some of my favorite comedies that spoof a particular genre. "Pandemonium" is not up there with those films. Most of the scenes in this movie had me sitting there in stunned silence because the movie wasn\'t all that funny. There are a few laughs in the film, but when you watch a comedy, you expect to laugh a lot more than a few times and that\'s all this film has going for it. Geez, "Scream" had more laughs than this film and that was more of a horror film. How bizarre is that?

*1/2 (out of four)'
Label 0
Review b"David Mamet is a very interesting and a very un-equal director. His first movie 'House of Games' was the one I liked best, and it set a series of films with characters whose perspective of life changes as they get into complicated situations, and so does the perspective of the viewer.

So is 'Homicide' which from the title tries to set the mind of the viewer to the usual crime drama. The principal characters are two cops, one Jewish and one Irish who deal with a racially charged area. The murder of an old Jewish shop owner who proves to be an ancient veteran of the Israeli Independence war triggers the Jewish identity in the mind and heart of the Jewish detective.

This is were the flaws of the film are the more obvious. The process of awakening is theatrical and hard to believe, the group of Jewish militants is operatic, and the way the detective eventually walks to the final violent confrontation is pathetic. The end of the film itself is Mamet-like smart, but disappoints from a human emotional perspective.

Joe Mantegna and William Macy give strong performances, but the flaws of the story are too evident to be easily compensated."
Label 0
Review b'Great documentary about the lives of NY firefighters during the worst terrorist attack of all time.. That reason alone is why this should be a must see collectors item.. What shocked me was not only the attacks, but the"High Fat Diet" and physical appearance of some of these firefighters. I think a lot of Doctors would agree with me that,in the physical shape they were in, some of these firefighters would NOT of made it to the 79th floor carrying over 60 lbs of gear. Having said that i now have a greater respect for firefighters and i realize becoming a firefighter is a life altering job. The French have a history of making great documentary\'s and that is what this is, a Great Documentary.....'
Label 1
Label 0 corresponds to neg
Label 1 corresponds to pos
Found 25000 files belonging to 2 classes.
Using 5000 files for validation.
Found 25000 files belonging to 2 classes.

코드 설명

  • 필요한 패키지들을 임포트하고 별칭(alias)을 지정합니다.

  • IMDB 데이터를 불러와 Colab이 접근할 수 있는 위치에 저장합니다.

  • 원본 데이터의 샘플 하나를 콘솔에 출력하여 확인합니다.

  • 원본 데이터를 학습(train) 데이터셋과 테스트(test) 데이터셋으로 분할합니다.

  • 분할된 학습 데이터를 사용해 모델을 구축합니다.

  • 주어진 데이터가 부정적 리뷰(neg)인지 긍정적 리뷰(pos)인지 분류하려 시도합니다.