do I need to clear batch data after processing it, from GPU memory in Pytorch?

Hey, I'm new to PyTorch and I'm doing a cat vs dogs on Kaggle. So I created 2 splits(20k images for train and 5k for validation) and I always seem to get "CUDA out of memory". I tried everything, from greatly reducing image size (to 7x7) using max-pooling to limiting the batch size to 2 in my dataloader. I always seem to use up all the memory after only one epoch.

That makes me wonder, I'm I required to clear batches from memory after I'm done training with the batch? If so how?

Here is my kaggle notebook, if that is of any use - https://www.kaggle.com/default404/dogvcatori

Any help is appreciated as im stuck with this for over 1 day now.

πŸ‘︎ 6
πŸ’¬︎
πŸ“…︎ Jan 07 2022
🚨︎ report
PancakeDB, a new type of database for incremental/Streaming writes and batch processing reads github.com/pancake-db/pan…
πŸ‘︎ 20
πŸ’¬︎
πŸ‘€︎ u/mwlon
πŸ“…︎ Jan 11 2022
🚨︎ report
This week we've been processing the cold chain Japanese Sake batch and setting up a country wide offline and online consumer study to gauge wider demand for premium product. This work is done within the framework of the Japan's M. of Agriculture led export consortium. $Wabi. https://t.co/JKxMxjT1XZ twitter.com/wabitoken/sta…
πŸ‘︎ 7
πŸ’¬︎
πŸ‘€︎ u/alexbus
πŸ“…︎ Jan 06 2022
🚨︎ report
Lambda function to use S3 batch processing to convert all objects to lowercase?

I am migrating an IIS web setup that is serving mainly pdf files to s3/cloudfront. I ran into the issue of mixed case objects in s3. I created a cloudfront function to convert all incoming request uri's to lowercase, I just need to get the objects themselves to lowercase. I was going to use s3 batch operations, but the closest example I could find works off PUT requests to the bucket and wont work wit this use case. I was trying to combine it with one of the AWS s3 batch lambda examples, but this is a bit beyond my current knowledge.

πŸ‘︎ 5
πŸ’¬︎
πŸ‘€︎ u/mr_mgs11
πŸ“…︎ Nov 22 2021
🚨︎ report
[Question] Batch processing images. Trouble with Python cv2.imwrite?

My code: https://pastebin.com/vSQpus3Y

I have 3 images as input. I've created a script to process those 3 images with an unsharp mask. It seems that cv2.imwrite is not wanting to output my images once I add a converted str from a variable to the filename. str(file_name) being it. The commented out print statements were to see if each variable was received correctly in each loop. Getting rid of str(file_name) from cv2.imwrite results in only one of three images being output.

πŸ‘︎ 2
πŸ’¬︎
πŸ‘€︎ u/Ridog8
πŸ“…︎ Dec 05 2021
🚨︎ report
Batch Processing Commercials

Hi All. Apologies in advance for being both a radio and reddit newbie. I work at a small market FM station which broadcasts a popular daily syndicated show from Premiere Networks. Once per week, we download all the show's commercials for the week from Premiere.

The hundred or so individual audio files are then processed in TLC to add cart numbers and EOM, and then are saved/sent to the appropriate location.

Currently, we process the files one-by-one. Have you developed a way to automate this process, and/or process the files as a batch? TIA for your help!

πŸ‘︎ 2
πŸ’¬︎
πŸ‘€︎ u/RestBig3731
πŸ“…︎ Dec 06 2021
🚨︎ report
Does Bitwig have the ability to run batch jobs for processing a large number of samples?

I'm specifically looking for a way to time stretch samples in bulk.

I currently have a way to do this from the command line on macOS/Linux or in python, using the rubberband library or command, but the time stretching in Bitwig is so much higher quality! I was hoping to find a way to process thousands of samples using the Elastique Pro algorithm. If anyone has any tips on how to do this, or something similar, I would be very appreciative!

I'm expecting that this won't be possible, but maybe I'll be surprised :)

πŸ‘︎ 2
πŸ’¬︎
πŸ“…︎ Nov 23 2021
🚨︎ report
Batch processing... reddit.com/gallery/r78727
πŸ‘︎ 19
πŸ’¬︎
πŸ‘€︎ u/placeb021
πŸ“…︎ Dec 02 2021
🚨︎ report
Poor quality output songs and batch processing

I have been playing with this recently and it's surprisingly good. I'm really enjoying being able to play beat saber to songs that I really enjoy but most people would never consider making a map for.

That said, the output file is massively compressed to the point that I can hear compression artifacts in my Oculus CV1. I can fix this by adding the same amount of silence to a high quality file and swapping out the song.ogg file, but that's quite a pain in the ass.

If I were to become a patron, will that output higher quality song files? If not, is there any chance this feature can be added?

Also, what are the chances we can get some batch processing of songs?

πŸ‘︎ 3
πŸ’¬︎
πŸ‘€︎ u/rrawk
πŸ“…︎ Dec 10 2021
🚨︎ report
Nice greed dildo today...so my understanding is Computershare does a batch processing of our purchased shares around 10:20 to 10:30 EST daily.
πŸ‘︎ 63
πŸ’¬︎
πŸ‘€︎ u/GrapeApeTheGreat
πŸ“…︎ Sep 27 2021
🚨︎ report
New Colab notebook "Batch image VQGAN+CLIP- public" by dan(at)dannyperry. This notebook allows processing of every frame of an initial video with VQGAN+CLIP. twitter.com/artificial_ar…
πŸ‘︎ 16
πŸ’¬︎
πŸ‘€︎ u/Wiskkey
πŸ“…︎ Oct 16 2021
🚨︎ report
Batch processing, restarting macro, keeping array, global array/global table

Hello!

I am batch processing a huge number of files. My macro structure is the following:

I start the batch macro, it opens one image after another and runs two separate macros on each of these.

For each image, different parameters are stored in arrays, exported as csv etc.

Now I would like to create one array that is populated with parameters from EACH image by the macro that is run on each each image by the batch macro.

How can I create and populate global arrays in imageJ?

Thank you!

πŸ‘︎ 2
πŸ’¬︎
πŸ‘€︎ u/papalagirauscher
πŸ“…︎ Nov 23 2021
🚨︎ report
Help batch processing!

Hi guys!

I'm trying to batch process seasons of episodes of a TV show. It's animation. Once I set the episode's video (h264, 576p, animation profile, RF26), that stays from episode to episode. However, the audio tracks...well, they don't. I need to manually change the transcoding I need to do for every single track for every single episode, even if the transcoding is always the same (I want output at AAC Stereo 64kbps). Same for subtitles! I need to add them all, all passthrough, on every single episode! Otherwise it chooses hardcoded first track by itself for subs, and the first audio track passthrough, or AAC 160Kbps, whichever it feels like). Is there a way to change both audio and subs choice for EVERY episode in a season, all at once?

My job processing usually involves adding a folder, then clicking on each episode, change audio, change subs, click add to queue, then click on next episode on the list. I'd like to just choose the same settings for all of them...How can I do this?

Thanks!

πŸ‘︎ 5
πŸ’¬︎
πŸ‘€︎ u/iturnedintoanewt
πŸ“…︎ Oct 14 2021
🚨︎ report
Canon DPP batch processing

I just installed Canon Digital Photo Professional software (RAW developer) on windows 7.

I am looking to develop around 100 raw files at a time and I saw there is a batch processing option under files menu. But I don't see how I can apply settings on all the Raw images for batch processing, by just setting one of them? For example right now I am looking at changing white balance of all the images to daylight. Do I have to adjust settings for example every single raw file?

πŸ‘︎ 2
πŸ’¬︎
πŸ‘€︎ u/isentropicfox
πŸ“…︎ Oct 19 2021
🚨︎ report
Batch export multiple clips on different tracks sharing aux processing - as individual files?

Wonder if you can help.

I have 50 different SFX clips I have created, one on each track (so 50 tracks), and they are sent into an aux that has some reverb and delay.

I would like to export these individually - including the aux processing - resulting in 50 separate files all with the aux processing.

If I use COMMIT it obviously only commits the Insert processing (of which I have none).

command-shift_k only exports the clips without any processing.

What is the best way to get all these clips processed and exported as individual files?

Thanks!

PT Ultimate 2021.3 - Mac OS Catalina

πŸ‘︎ 5
πŸ’¬︎
πŸ‘€︎ u/TheMoogerfooger
πŸ“…︎ Sep 13 2021
🚨︎ report
Despite my explicit instructions not to, my husband tightened the bands after processing 5 pints of corn. Do I need to freeze this batch now?

I was getting tired last night and my husband offered to wait for the canner to depressurize and take out the jars. He asked if he should tighten the bands. I said no, don't touch the bands.

And this morning he tells me he tightened all the bands because they were pretty loose.

Can I trust these seals or do I need to freeze this batch? I'm not interested in another 3 hour processing time suck. (Pressure canned at 11lbs for 55 minutes + warm up + depressurize time)

πŸ‘︎ 11
πŸ’¬︎
πŸ‘€︎ u/jsat3474
πŸ“…︎ Aug 25 2021
🚨︎ report
New Colab notebook "Batch image VQGAN+CLIP- public" by dan(at)dannyperry. This notebook allows processing of every frame of a video with VQGAN+CLIP. twitter.com/artificial_ar…
πŸ‘︎ 11
πŸ’¬︎
πŸ‘€︎ u/Wiskkey
πŸ“…︎ Oct 16 2021
🚨︎ report
Batch processing of detection data

Hello all,

Quick question that I am hoping someone can help me with. To put it simply, what is everyone using to process their detection data?

I have participated in several undergraduate research projects at my university, and one thing I've noticed is that my collegues are happy to process their data manually, in excel or origin. Maybe I'm just lazy but I feel as though automating these tasks would make my life a hell of a lot easier. For this particular project we're comparing UV/Vis spectra of food samples, across several wavelengths. I'm required to split a single column .asc file into portions based on a heading and add a column to each file. After some more processing I need to visually compare several different spectra to mark out related peaks. So after my 30th hour of manually working up a data set, and fruitless googling, I thought I would turn to you, oh esteemed labrats. What do you use?

I have toyed with Origin and excel Power Query, and I've done some undergrad C++. I'm sure I could write a program to at least break the files up and add the second column. However, this is more time invested in a direction that could prove fruitless.

I would be satisfied if there is some easily accessible resource that someone could point me towards to, I am completely lost right now.

Thanking you in advance!

πŸ‘︎ 3
πŸ’¬︎
πŸ‘€︎ u/cookiecutter73
πŸ“…︎ Sep 07 2021
🚨︎ report
ReelSteady GO Batch Processing using AutoHotKey Version 2

Found an old AHK script by @paytonfrost and made a few minor tweaks to it. It's still very sledgehammery, but maybe someone will find it useful or build upon it. Two scripts - first batch smoothes the files using ReelSteady GO, second puts the smoothed files in their own folder and renames them to the original file name (using PowerShell on Windows) - good for relinking media in editing software. Compared to @paytonfrost 's original, I've added keywaits for the syncing and rendering times: you can speed up the process by tapping "a" as soon as the ReelSteady has done its thinking and is waiting for input. Also added some janky adjustments to the smoothness slider as I prefer it reduced a bit. Reelsteady GO code:

    #SingleInstance Force
    CoordMode, Screen

    InputBox, FileNum, Reelsteady Batch, How many files?, ,60    ; how many files are in the sd card? 
																; note this will convert all the video files in the card

     ;Variable Declaration
    i = 1
    LongestLength = 60  							;length of longest clip to process, in seconds
    ScrollAmount := (i - 1) * 2

    sleep, 1000

    if (errorlevel = 0){
        msgbox, 4, screens, is second screen unplugged? 	; multiple screens can mess up pixel positions
        IfMsgBox, No 										; unplug second screen and relaunch script
	        Return
        InputBox, LongestLength, Reelsteady Batch, Longest file mins - roundup, ,60    ; how long is the longest clip in the card in mins?
	        if (errorlevel = 0){
	        RenderTime := LongestLength*60*2.5				; these threee equations work for my system. adjust as you see fit
	        ProcessTime1 := 16+LongestLength*60/5
	        ProcessTime2 := 10+LongestLength*60/10
	        Run, C:\Program Files\ReelSteadyGo\ReelSteadyGo.exe
	        sleep, 6000
	        send #{up}										;maximise window - keeps pixels in right places
	        Loop {
		        sleep, 1000
		          Winactivate ReelSteady GO
		        Click 109,1012						  ;Load video button - change to appropriate pixel position
		        sleep 1000
		        Click 115,1012
		        sleep, 1000
		        Send +{TAB 2}
		        Send {End} 
		        sleep, 1000
		        Send {Enter} 
		        tooltip % i
		        sleep, 1000
		        Send {End} {Home}
		        ScrollAmount := (i - 1)*2			  ;select the next file
... keep reading on reddit ➑

πŸ‘︎ 3
πŸ’¬︎
πŸ‘€︎ u/whereistomba
πŸ“…︎ Sep 18 2021
🚨︎ report
Batch Processing

Hi, is there some package on go that has the same capabilities of spring batch? I trying to find some but no luck.

πŸ‘︎ 4
πŸ’¬︎
πŸ‘€︎ u/jcccoquia
πŸ“…︎ Sep 02 2021
🚨︎ report
I'm trying to do batch processing, but the complexities are beyond me.

Hi all,

I'm working with JPGs that require conversion to CMYK, the Exposure BlowUp plugin to be run at 300%/900dpi, then saved as TIF with ZIP compression. According to the person who taught me this procedure, who has a great deal of experience with comics restoration in the way I'm dealing with, making this a process done in multiple steps can reduce photo quality.

Now I can do this manually, but it takes a minute and a half and I have over 10,000 files to get through. The person who explained the process to me said he couldn't find a way to do it in batch format, but I was hoping maybe reddit could point me in the right direction so I wasn't doing this until the day I die.

I'll answer whatever questions make this easier, but I don't know what information you need. All I know is that right now, I'd really like to do something more productive than do this for every single page click by click. Despite how it sounds, I do have a life outside of this. Please help me if you can, I will take any and all thoughts.

πŸ‘︎ 2
πŸ’¬︎
πŸ‘€︎ u/CMYKAvenger
πŸ“…︎ Oct 08 2021
🚨︎ report
Processing my first batch

Hi all, I’ve been looking into this for a few months and collecting amanitas and drying them to store, planning to use the two step dehydration and boil method. However I’ve since heard that there’s new scientifically verified information stating that it is better to use fresh mushrooms and boil them and do the conversion completely in the boiling pot. I just found a large amount of fresh amanitas yesterday by chance so would like to try this method. I struggled to sift through the scientific articles about it as I don’t understand all the language. Just want to know how many of you have used this method and can vouch for it.

Also is there an amount of lemon juice or citric acid you use depending on the ratio of water or mushrooms? I’ll buy a ph kit too as I think that’s safer but generally wondering if there is an estimated amount you start with.

Also what is the boiling time and temp you have all had success with.

Thank you very much to anyone who can provide their successful experiences.

TLDR

  1. Have you boiled raw amanitas to get best ibo to muscimol conversion,
  2. Estimate amount of acid needed,
  3. What is the boiling time and temp
πŸ‘︎ 11
πŸ’¬︎
πŸ‘€︎ u/lucidlucifer94
πŸ“…︎ Jul 04 2021
🚨︎ report
From my first try batch processing E-6 (Fuji GA645W // Fuji Velvia 100 // Unicolor E6 Kit)
πŸ‘︎ 102
πŸ’¬︎
πŸ‘€︎ u/rumpgump
πŸ“…︎ Aug 05 2021
🚨︎ report
GUI ways to Select + Batch process/ convert formats Videos in folder trees? UI for Selection & Processing visibility on small collections, instead of a BATCH or SCRIPT running blind behind the scenes.

GUI ways to Select + Batch process/ convert formats Videos in folder trees?

UI for Selection & Processing visibility on small collections, instead of a BATCH or SCRIPT running blind behind the scenes.

  • But I'd like to preferably run a BATCH of VIDEOS in Folder Trees through this process.
  • I will be doing it "small collections" at a time, would like to have visibility instead of a BATCH or SCRIPT running somewhere and having to be modified for every such collection.
  • Current INITIAL collections are FLV to MP4 - Some others may be different formats/ codecs later.
  • Hence VISIBILITY of INPUT folders/ files and OUTPUT would be nice to keep track of.
  • Ideally with UI that "shows the list of Selected Source videos" before the BATCH PROCESSING; and maybe during/ after.

Might have to be doing various "FLAVORS" of conversions of X format to Y format.. in coming weeks/ months. But VISIBILITY & CONFIGURABILITY is key in the UI.

What I need:

So GUIs for FFMeg that have Batch with VISIBILITY or Other non FFMPEG - Batch processors with UI with Selection Visibility.

---------------

First collections:

FLV to MP4 Commandline w/o Reencode for FLVs of current collection V: .H264/ AVC A: AAC/ mp4a

From these threads it seems, its better to copy streams from container without "conversion".

Method seems simple enough via FFMpeg for a single video.

https://obsproject.com/forum/threads/guide-how-to-convert-flvs-to-mp4-fast-without-re-encoding.6406/

>ffmpeg -i %1 -c copy -copyts %1.mp4[/CODE]

https://obsproject.com/forum/resources/how-to-convert-flvs-to-mp4-fast-without-re-encoding.78/

>ffmpeg -i input.flv -c copy -copyts output.mp4
>
>The "-c copy" means it will just copy the audio and video tracks without re-encoding them.
>
>The "-copyts" flag means it will copy timestamps, which should help with syncing audio and video.

https://obsproject.com/forum/resources/guide-batch-convert-flv-to-mp4-losslessly-with-ffmpeg.525/

echo off

echo =========================================

echo Flv to Mp4 batch script using ffmpeg v1.0

echo =========================================

echo.

`echo Press any

... keep reading on reddit ➑

πŸ‘︎ 3
πŸ’¬︎
πŸ‘€︎ u/alexaxl
πŸ“…︎ Aug 03 2021
🚨︎ report
Cyber criminals move faster than ML and AI built on legacy batch processing. Here’s the modern solution for cybersecurity: data stream processing. quix.ai/cyber-security-ma…
πŸ‘︎ 5
πŸ’¬︎
πŸ‘€︎ u/Quix_ai
πŸ“…︎ Sep 21 2021
🚨︎ report
Registax 6 Batch Processing Tutorial
πŸ‘︎ 2
πŸ’¬︎
πŸ‘€︎ u/Ioan_Nemes
πŸ“…︎ Sep 25 2021
🚨︎ report
Batch Processing for Frames in a Time lapse

I've got a few hundred images that I took over the weekend that I want to splice together into a time lapse. However, I don't particularly relish having to process 300 images manually. Does anybody know of a good way to perform some processing steps on a reference image from the sequence, and then batch those to the rest of the images for consistency and sanity sake, before I add them all together into a video?

πŸ‘︎ 2
πŸ’¬︎
πŸ‘€︎ u/astrodj2112
πŸ“…︎ Aug 17 2021
🚨︎ report
Anyone know how to do batch processing with remini?

I know you could use bluestacks to access remini but it's not super great for processing multiple images. Anyone know of an api to do this?

πŸ‘︎ 5
πŸ’¬︎
πŸ‘€︎ u/jordanearth
πŸ“…︎ Jul 21 2021
🚨︎ report

Please note that this site uses cookies to personalise content and adverts, to provide social media features, and to analyse web traffic. Click here for more information.