View Issue Details

IDProjectCategoryView StatusLast Update
0000358VisualDifferFolders differpublic2023-07-23 04:19
Reportertkmantis Assigned Toadmin  
PrioritynormalSeverityfeatureReproducibilityN/A
Status assignedResolutionopen 
Product Version1.8.7 
Summary0000358: Proposal for Efficient Compare over network
DescriptionI often use Visual Differ to compare a local folder on one Mac to a folder on another MAC on my LAN (Local Area Network) hosting an AFP or SMB share. I find the scan/compare of the file metadata is much slower over the network. I believe this is because when Visual Differ walks the directory trees it sends many small network requests to the Mac hosting the file share to query the file attributes of each file such as date, size, tags, labels, etc. When there are thousands of files to hundreds of thousands of files, this time adds ups, even if done over Ethernet instead of Wifi.

It would be nice if a compare over the network could work more efficiently as follows:

1) Visual Differ would be enhanced to include a background process that is able to receive requests from other copies of Visual Differ running on the same LAN.
2) If user starts a compare and Visual Differ detects that a folder to be compared is on another MAC on the LAN (SMB or AFP share)...
3) Check if that Mac hosting the file share also has Visual Differ software installed
4) If the Hosting Mac has Visual Differ software installed, the Mac performing the compare sends a request to the background process running in the Hosting Mac, indicating a folder on the share.
5) The background process would enumerate the contents of that folder creating a list of all files, sizes, dates, labels, resource sizes, etc and create a data blob to send over the LAN as a single object back to the Visual Differ instance that needs it.
6) The Visual Differ running the compare can then use the blob of data received in step 5 for all comparison matching tasks, making new network requests only if file content comparison is requested, or when a user acts on the result of the comparison to copy or delete files.

Thanks
TagsNo tags attached.

Activities

tkmantis

2023-02-20 07:18

reporter   ~0000744

Just wondering if this might become a reality? Any status update? Thanks!

admin

2023-02-20 07:36

administrator   ~0000745

After some tests was impossible to reproduce the problem also configuring remote networks so I given up
If I can't reproduce the problem I can't find any solution, and all code related to file system use Apple public API so it's very difficult to understand where apply some modification

I leave the bug open because I'm sure the problem exists

tkmantis

2023-02-20 21:52

reporter   ~0000746

I actually do not consider this a bug, but a feature request for a feature I really would like to see, a major algorithm improvement opportunity for dealing with comparison over a network....

I believe the comparison process over a network would be capable of running much faster if you built the feature as I described above.

You should be able to see how network comparisons are slower than local comparisons if you set up the following test:
1) Set up 2 USB external hard drives, each with an identical large collection of files, perhaps 100,000 files.
2) Plug both hard drives into one Mac and run a comparison of file timestamps, sizes, and finder tags (but NOT including the option for file content). Note the time taken from the log console when no network is involved.
3) Now plug one of the hard drives into another Mac and file share it to the first mac running Visual Differ and then perform the comparision again and note the time taken in the log console with a network involved.
4) When you try this test, please describe your scenario - The number of files you set up, the time taken in step 2, time taken in step 3, the type of network connection use used.

[Of course my real life scenario is one data set is on the internal drive of the Mac doing file sharing. For two external hard drives I would connect them to the same machine to compare. I am suggesting you do 2 USB hard drives as it is something you could easily setup to reproduce my observation.]

tkmantis

2023-07-23 04:19

reporter   ~0000751

I am following up, wondering if you had any further thoughts on how this proposed feature could become a reality? Thanks

Issue History

Date Modified Username Field Change
2022-11-10 03:21 tkmantis New Issue
2022-11-10 03:21 tkmantis Status new => assigned
2022-11-10 03:21 tkmantis Assigned To => admin
2023-02-20 07:18 tkmantis Note Added: 0000744
2023-02-20 07:36 admin Note Added: 0000745
2023-02-20 21:52 tkmantis Note Added: 0000746
2023-07-23 04:19 tkmantis Note Added: 0000751