php - Uploading large (~80Mb) text file and saving to database? -

-

i need create web page person can use select , upload text file (eg: csv) , have file imported backend mysql database.

this fine, , have php code - accept file upload, parse, save database.

the problem need reliable method allow user upload large files, potentially around 30 - 80mb, without timing out or failing during upload or save database.

other increasing upload limit , timeout settings, there other advice on how can ensure consistently reliable method ensure data uploaded?

the dataset changes daily, , previous day's data in database needs cleared out , replaced new day's data - vital no data lost during new upload.

i welcome advice of others how best achieve this, high degree of confidence data uploaded reliably , successfully, , how best clearing out of previous days data - ideally best check new data uploaded before clearing out , replacing previous data.

all suggestions welcome!

there stack overflow question confirm goleztrol's answer php timeout doesn't begin until after upload finished: php file upload affected or not max_input_time?

as processing data, if file in csv/tab-delimited or similar format matches table structure, fastest method use mysql's load data statement: https://dev.mysql.com/doc/refman/5.5/en/loading-tables.html

for aspect of replacing data , making sure import successful before deleting old data, best answer question: mysql: truncate table within transaction? talks how can inserting data new table, once you've verified new data, renaming tables swap them. can safely delete old data. seems fluid way of accomplishing (keeping in mind truncate not transaction-safe).


Comments

Post a Comment

Popular posts from this blog

python - TypeError: start must be a integer -

-
i trying draw 5 turtles, getting typeerror on line 25. here code: import turtle wn = turtle.screen() redrose = turtle.turtle() color = input("what background color be?") fillcolor_f = input("what color of rose be?") redrose.hideturtle() redrose.speed(30) redrose.penup() redrose.left(180) redrose.forward(175) redrose.right(90) redrose.forward(30) redrose.right(90) redrose.pendown() def drawrose(red): redrose.color("pink") redrose.fillcolor(fillcolor_f) redrose.fill(true) in range(red): redrose.forward(i) redrose.right(49) in range(5): drawrose(redrose) redrose.penup() redrose.forward(350) redrose.right(144) redrose.pendown() redrose.fill(false) drawrose(50) wn.bgcolor(color) i trying draw 5 roses, produces errors. doing in interactivepython.org. you recursively calling drawrose wrong parameter. on line 23 ( for in range(red): ) expect red integer...
Read more

c# - DevExpress RepositoryItemComboBox BackColor property ignored -

-
i have following code almost works properly; private void cbstatus_drawitem(object sender, listboxdrawitemeventargs e) { string item = e.item string; if (item != null) { switch (item) { case "1": e.appearance.forecolor = color.green; e.appearance.backcolor = color.green; break; case "2": e.appearance.forecolor = color.orange; e.appearance.backcolor = color.orange; break; case "3": e.appearance.forecolor = color.red; e.appearance.backcolor = color.red; break; } } } when dropdown shown, items forecolor correct, backcolor remains whatever backcolor of theme is; i.e. if i've got set dark theme, backcolor dark, cells in gridview, rather being green/orange/red. i've tried setting e.appearance.options.usebackcolor trying set e....
Read more

django - Creating multiple model instances in DRF3 -

-
1) create multiple model instances 1 api call, asked here: how create multiple model instances django rest framework? tried solution named in link, no success. i trying upload multiple files in 1 api call. result: files uploaded (only once overwrote perform_create) 1 instance created (if send 2 files, latter created instance). my code: class fileuploadserializer(serializers.modelserializer): def __init__(self, *args, **kwargs): many = kwargs.pop('many', true) super(fileuploadserializer, self).__init__(many=many, *args, **kwargs) class meta: model = fileupload read_only_fields = ('created', 'datafile', 'owner') class fileuploadviewset(viewsets.modelviewset): queryset = fileupload.objects.all() serializer_class = fileuploadserializer parser_classes = (multipartparser, formparser, ) def perform_create(self, serializer): file_list = self.request.data.getlist('file') ...
Read more