Python根據URL地址下載檔案並儲存至對應目錄

引言

在程式設計中經常會遇到圖片等資料集將圖片等資料以URL形式儲存在txt檔案中，為便於後續的分析，需要將其下載下來，並按照資料夾分類儲存。本文以Github中Alexander Kim提供的圖片分類資料集為例，下載其提供的圖片樣本並分類儲存
Python 3.6.5，Anaconda， VSCode

1. 下載資料集檔案

建立專案資料夾，下載上述Github專案中的raw_data資料夾，並儲存至專案目錄中。
## 2. 獲取樣本檔案位置
編寫get_doc_path.py，根據根目錄位置，獲取目錄及其子目錄所有資料集檔案

import os


def get_file(root_path, all_files={}):
    '''
    遞迴函數，遍歷該檔案目錄和子目錄下的所有檔案，獲取其path
    '''
    files = os.listdir(root_path)
    for file in files:
        if not os.path.isdir(root_path + '/' + file):   # not a dir
            all_files[file] = root_path + '/' + file
        else:  # is a dir
            get_file((root_path+'/'+file), all_files)
    return all_files


if __name__ == '__main__':
    path = './raw_data'
    print(get_file(path))

3. 下載檔案

3.1 讀取url列表並

for filename, path in paths.items():
        print('reading file: {}'.format(filename))
        with open(path, 'r') as f:
            lines = f.readlines()
            url_list = []
            for line in lines:
                url_list.append(line.strip('\n'))
            print(url_list)

3.2 建立資料夾

foldername = "./picture_get_by_url/pic_download/{}".format(filename.split('.')[0])
if not os.path.exists(folder_path):
        print("Selected folder not exist, try to create it.")
        os.makedirs(folder_path)

3.3 下載圖片

def get_pic_by_url(folder_path, lists):
    if not os.path.exists(folder_path):
        print("Selected folder not exist, try to create it.")
        os.makedirs(folder_path)
    for url in lists:
        print("Try downloading file: {}".format(url))
        filename = url.split('/')[-1]
        filepath = folder_path + '/' + filename
        if os.path.exists(filepath):
            print("File have already exist. skip")
        else:
            try:
                urllib.request.urlretrieve(url, filename=filepath)
            except Exception as e:
                print("Error occurred when downloading file, error message:")
                print(e)

4. 完整原始碼

4.1 get_doc_path.py

import os


def get_file(root_path, all_files={}):
    '''
    遞迴函數，遍歷該檔案目錄和子目錄下的所有檔案，獲取其path
    '''
    files = os.listdir(root_path)
    for file in files:
        if not os.path.isdir(root_path + '/' + file):   # not a dir
            all_files[file] = root_path + '/' + file
        else:  # is a dir
            get_file((root_path+'/'+file), all_files)
    return all_files


if __name__ == '__main__':
    path = './raw_data'
    print(get_file(path))

4.2 get_pic.py

import get_doc_path
import os
import urllib.request


def get_pic_by_url(folder_path, lists):
    if not os.path.exists(folder_path):
        print("Selected folder not exist, try to create it.")
        os.makedirs(folder_path)
    for url in lists:
        print("Try downloading file: {}".format(url))
        filename = url.split('/')[-1]
        filepath = folder_path + '/' + filename
        if os.path.exists(filepath):
            print("File have already exist. skip")
        else:
            try:
                urllib.request.urlretrieve(url, filename=filepath)
            except Exception as e:
                print("Error occurred when downloading file, error message:")
                print(e)


if __name__ == "__main__":
    root_path = './picture_get_by_url/raw_data'
    paths = get_doc_path.get_file(root_path)
    print(paths)
    for filename, path in paths.items():
        print('reading file: {}'.format(filename))
        with open(path, 'r') as f:
            lines = f.readlines()
            url_list = []
            for line in lines:
                url_list.append(line.strip('\n'))
            foldername = "./picture_get_by_url/pic_download/{}".format(filename.split('.')[0])
            get_pic_by_url(foldername, url_list)

4.3 執行結果

執行get_pic.py
當程式意外停止或再次執行時，程式會自動跳過資料夾中已下載的檔案，繼續下載未下載的內容

{‘urls_drawings.txt’: ‘./picture_get_by_url/raw_data/drawings/urls_drawings.txt’, ‘urls_hentai.txt’: ‘./picture_get_by_url/raw_data/hentai/urls_hentai.txt’, ‘urls_neutral.txt’: ‘./picture_get_by_url/raw_data/neutral/urls_neutral.txt’, ‘urls_porn.txt’: ‘./picture_get_by_url/raw_data/porn/urls_porn.txt’, ‘urls_sexy.txt’: ‘./picture_get_by_url/raw_data/sexy/urls_sexy.txt’}
reading file: urls_drawings.txt
Try downloading file: http://41.media.tumblr.com/xxxxxx.jpg
Try downloading file: http://41.media.tumblr.com/xxxxxx.jpg
Try downloading file: http://ak1.polyvoreimg.com/cgi/img-thing/size/l/tid/xxxxxx.jpg
Error occurred when downloading file, error message:
HTTP Error 502: No data received from server or forwarder
Try downloading file: http://akicocotte.weblike.jp/gaugau/xxxxxx.jpg
Try downloading file: http://animewriter.files.wordpress.com/2009/01/nagisa-xxxxxx-xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg

後注：由於樣本資料集內容的問題，上述地址以xxxxx代替具體地址，案例專案也已經失效，但是方法仍然可以借鑑