View Issue Details

IDProjectCategoryView StatusLast Update
0000218VisualDifferGeneralpublic2014-02-23 09:27
ReporterMr.KiwiAssigned Toadmin 
PriorityhighSeveritymajorReproducibilityhave not tried
Status closedResolutionfixed 
PlatformMacOSMavericksOS Version10.9.1
Product Version1.5.7 
Target VersionFixed in Version1.5.9 
Summary0000218: Copy and Sync folders set invalid file time on samba volumes
DescriptionAction "copy to right" and "sync to right" not respect the origin file timestamp and not set the timestamp well.
Steps To ReproduceAssumption:
- On left side a folder with some files exists
- On right side this folder and the content is complete missing

than
- select folder on left side an use action "copy to left" or "sync to right"
- as result all files and folders will be copy, but all file time stamps are set invalid (current time)

- As result, all files/folders will be shown as different
TagsNo tags attached.

Activities

admin

2013-12-31 08:11

administrator   ~0000424

Which type of filesystems (HFS, NTFS, FAT32) are on left *AND* right?
Are disk removable?

On my mac formatted with (HFS case insensitive) everything works fine

Please give me more details because I can't reproduce this bug

Mr.Kiwi

2014-01-01 14:32

reporter   ~0000425

The filesystem is an local mounted NAS, smbfs. With early version of VisualDiff (and MacOS 10.7), "Sync" has set timestamp well, but "Copy" not.

admin

2014-01-01 18:23

administrator   ~0000426

Ok, I never tested with smbfs this should be the problem.

"Sync" internally calls "copy" so it is very strange these actions works differently, I will try to reproduce the bug

thanks

admin

2014-01-02 07:55

administrator   ~0000427

OK, I can confirm randomly the date/time isn't set.

The bug isn't strictly related to VisualDiffer but how Apple system calls work, I'm searching a workaround.

Please apologize for this inconvenient

Mr.Kiwi

2014-01-02 10:56

reporter   ~0000428

First)
Happy New Year and best greetings from Germany!

Second)
I test follow in a terminal:

my-mac: kiwi$ mount
...
//GUEST:@my-nas/Public on /Volumes/Public (smbfs, nodev, nosuid, noowners, mounted by kiwi)

my-mac: kiwi$ cd /Volumes/Public/Sample

my-mac: kiwi$ touch readme.txt

my-mac: kiwi$ ls -l
-rwxrwxrwx 1 kiwi staff 0 2 Jan 2014 readme.txt

my-mac: kiwi$ touch -t 1206232233.44 readme.txt

my-mac: kiwi$ ls -l
-rwxrwxrwx 1 kiwi staff 0 23 Jun 2012 readme.txt

So, you can see, that the unix command "touch" can set the timestamp well. May be, you should use "uptime()" to set the timestamp.

Mr.Kiwi

2014-01-02 10:58

reporter   ~0000429

Sorry, my mistake: I mean "utime()" not "uptime()"

admin

2014-01-02 10:59

administrator   ~0000430

Hi, happy new year also to you :)

The bug is strange, if I use the VisualDiffer touch everything works fine (it does exactly the same code to change attributes).

The creation time is set correctly but the modification time is "reset" to the copy time but both dates are set in same call, really weird

admin

2014-01-02 11:23

administrator   ~0000431

Your hint is correct, using utimes (utime is deprecated) seems to work.
I suspect the cocoa API have some bug

I would ask if you can help me to test this fix, I would send to you an alpha version so you can test yourself.

Mr.Kiwi

2014-01-02 11:25

reporter   ~0000432

Yes, sure, I can test it.

admin

2014-01-02 11:36

administrator   ~0000433

At the link below you can find the version with the fix.

** Please consider this is an alpha **

1. Do not overwrite the Mac App Store version
2. Unzip where you prefer, also your home directory
3. run as usual
4. test it but don't do with important or not backed up data (I repeat this is an alpha)

Thanks again to help me on 2nd January :)

http://cl.ly/220f2f302H18

Mr.Kiwi

2014-01-02 14:35

reporter   ~0000434

I have execute a test (as std-user, without admin privilege), results:

my-mac: kiwi$ mount

//GUEST:@my-nas/sandbox on /Volumes/sandbox (smbfs, nodev, nosuid, noowners, mounted by testuser)


Next, I create on NAS two „folder-1“ and „folder-2“.
Next, I copy from Apple-Default-Document „About Downloads.pdf“-Bundle to „folder-1“


Now:
- I start VisialDiffer 1.5.7 (AppStore version)
- Than I check behavior of „copy to right“ and „sync“ - and yes, bug is reproducable

Next:
- I start new VisualDiffer -bugfix218 Version (result --> app start well)
- I check with „lsof“ - yes the application runs from correct path ($HOME)
- Now I check output of „About VisualDiffer“ dialog - report version 1.5.7

- Now I choose „compare file timestamps and size“
- Now I select on left side the folder and choose „copy to right“ action, as result I get
     - some files have new timestamps (About Downloads.lpdf/Contents/Resources/ca.lproj/InfoPlist.strings)
     - some files have an timestamp (+ 1 second) and
     - some files have the correct (old) timestamp (from left side)

- Now I double-check the result, I choose „Compare file content only“
     - as result, no difference are shown, I assume, copy operation works well, only timestamp is invalid

- Now I select „compare file timestamps and size“ again and clean content of „folder-2“
- Than select content of „folder-1“ and execute „sync to right“ action
     - same result like „copy to right“, some timestamps are incorrect.


Finally with Finder:
I choose an original file (left side) and show information with Finder about this file,
and I see that creation,access,modification timestamp are equal.

I choose an „sync“ file (right side) and show information with Finder about this file,
and I see that „creation“ time is correct (like left side), but „modification“ and last-access time are newer (invalid).


Additional I see:

my-mac: kiwi$ cd /Volumes/sandbox
my-mac: kiwi$ ls -la folder-1/About\ Downloads.lpdf/Contents/Resources/hr.lproj/
-rwxrwxrwx 1 testuser staff 299334 2 Jan 13:04 About Downloads.pdf
-rwxrwxrwx 1 testuser staff 111 2 Jan 13:04 InfoPlist.strings


ls -la folder-2/About\ Downloads.lpdf/Contents/Resources/hr.lproj/
-rwxrwxrwx 1 testuser staff 4096 2 Jan 13:47 ._About Downloads.pdf
-rwxrwxrwx 1 testuser staff 4096 2 Jan 13:47 ._InfoPlist.strings
-rwxrwxrwx@ 1 testuser staff 299334 2 Jan 13:47 About Downloads.pdf
-rwxrwxrwx@ 1 testuser staff 111 2 Jan 13:47 InfoPlist.strings

Conclusion:
Somewhere seams to attach some file and create "._<filename>".

admin

2014-01-02 14:46

administrator   ~0000435

Great test suite.

The version with fix is 1.5.7 I haven't changed the number because official updated from MAS could be no longer "visible"

".lpdf" are special files in Apple world, they are resource forks so I think copying them on Windows filesystem can produce strange behavior.

I try to make deeper tests and I'll keep you informed.

Thank you very much

admin

2014-01-02 15:06

administrator   ~0000436

About one second difference here also from finder does the same.

From Finder I copy a file from mac mini to samba mapped disk and the copied modification time file is less than one second

You can check using stat <filename>>

In my case

stat /Volumes/smbtest/sample01.txt

771751939 9849149156497250956 -rwxrwxrwx 1 dave staff 0 3 "Jan 2 00:00:00 2014" "Jan 1 07:11:32 2014" "Jan 1 07:11:32 2014" "Nov 9 13:18:44 2013" 16384 32 0 /Volumes/smbtest/sample01.txt

and on mac mini

stat /Users/dave/test_suite/smbtest/sample01.txt
16777217 6771460 -rw-r--r-- 1 dave staff 0 3 "Jan 2 14:59:02 2014" "Jan 1 07:11:33 2014" "Jan 2 07:11:41 2014" "Nov 9 13:18:45 2013" 4096 8 0 /Users/dave/trash/test_suite/smbtest/sample01.txt

May you confirm??? This seems a problem related to OSX not VisualDiffer

Mr.Kiwi

2014-01-02 16:15

reporter   ~0000437

Oh, difficult:

three points I found out:

------ first point --------------

Assume, /Volume/sandbox is a NAS (like Buffalo, Qsnap ...), mounted as SMB.

Try follow shell commands:

cd /Volumes/sandbox
rm -rf /Volumes/sandbox/folder-1
rm -rf /Volumes/sandbox/folder-2
mkdir -p /Volumes/sandbox/folder-2
mkdir -p /Volumes/sandbox/folder-1/subfolder-1
mkdir -p /Volumes/sandbox/folder-1/subfolder-2
touch /Volumes/sandbox/folder-1/subfolder-1/content.txt
touch /Volumes/sandbox/folder-1/subfolder-2/content.txt
echo "hello" > /Volumes/sandbox/folder-1/readme.txt
sleep 5
( cd /Volumes/sandbox/folder-1 && cp -Rp subfolder-1 subfolder-2 readme.txt ../folder-2 )

As result, VisualDiffer shown no diffs, all timestamps are preserved well by "cp -Rp ..."

------------- second --------------

Use follow command lines

cd /Volumes/sandbox
rm -rf /Volumes/sandbox/folder-1
rm -rf /Volumes/sandbox/folder-2
mkdir -p /Volumes/sandbox/folder-2
mkdir -p /Volumes/sandbox/folder-1/subfolder-1
mkdir -p /Volumes/sandbox/folder-1/subfolder-2
touch /Volumes/sandbox/folder-1/subfolder-1/content.txt
touch /Volumes/sandbox/folder-1/subfolder-2/content.txt
echo "hello" > /Volumes/sandbox/folder-1/readme.txt
sleep 5
( cd /Volumes/sandbox/folder-1 && rsync -av --exclude '.DS_S*' --exclude '._*' Folder-1/ folder-2 )

As result, VisualDiffer shown no diffs, all timestamps are preserved well by "rsync ..."


------------ last -------------

cd /Volumes/sandbox
rm -rf /Volumes/sandbox/folder-1
rm -rf /Volumes/sandbox/folder-2
mkdir -p /Volumes/sandbox/folder-2
mkdir -p /Volumes/sandbox/folder-1
cp -rf $(HOME)/About\ Download.lprf /Volumes/sandbox/folder-1
sleep 5
( cd /Volumes/sandbox/folder-1 && cp -Rp About\ Download.lpdf ../folder-2 )

Result:
All timestamps of directories are preserved, but timestamps of files inside the *.lpdf are different.
This behavior I see also on MacOS 10.5.


----- conclusion -----

rsync works correct in all cases.

cp -Rp works correct for directories and subfolders, but failed with the *.lpdf file too.

VisualDiffer also failed, if I copy/sync "subfolder-1" from left to right.


So I can't confirm, that it is a MacOS problem only. The man page for "cp" is scary about the handling of preserve directory/xattr etc. So I will file a bug to apple for the "cp -Rp *.lpdf" issue.

But for the other files (I test also with a bunch of *.pdf files) VisualDiffer failed, but cp and rsync works well.

admin

2014-01-02 16:21

administrator   ~0000438

Last edited: 2014-01-02 16:22

View 2 revisions

I'm trying to understand where is the bug but it seems in somewhere apple code, I've written a little snippet (a separated project) and works badly.

I suppose cp uses some hack

admin

2014-01-03 13:15

administrator   ~0000439

Just some little modification, I'm sure Cocoa classes refer to a wrong date (maybe a cached value), now I've modified the code to use *only* unix low level calls to determine and set modification time.

I know I'm stressing your patience but if you can test it again would be great.

The link below to new alpha

http://cl.ly/2u1M3W243U33

thanks in advance

Mr.Kiwi

2014-01-05 16:44

reporter  

screenshot-218-a2.png (265,120 bytes)

Mr.Kiwi

2014-01-05 17:00

reporter   ~0000440

Sorry for the delay.

I have test your version 218-a2.zip today. I use the same test like described before. But I'm sorry, your app failed and the bug are not fixed.

I attach a screenshot (screenshot-218-a2.png). Like you can see, some files have a correct timestamp.

Additional, I take a look, how your app works. May be, I found your problem.

For a correct behavior, you must first create the empty directory structure and touch the timestamps. (Look to the rsync behavior.) Then you must copy the files and touch there timestamps. Please note, if you copy a new file into an directory, then the OS modify the timestamp of the directory too. So, you must modify the timestamp again.

After all is finished, you directory (modify) timestamp should be the newest timestamp of all files, that are members of this directory (in relation to the member creation timestamp).

But you work with a recursive pipe, that may be the problem.

admin

2014-01-05 17:17

administrator   ~0000441

First of all thank you very much to spend your time to make alpha testing for me.

Folders timestamps are set after all children but when the files are on smbfs (I suppose when they aren't on local disks) something doesn't work.

I will dedicate to this bug more time in next days, thanks again for your great feedback

admin

2014-01-06 13:55

administrator   ~0000446

Good news, using a complex structure with many files now all modification timestamps are copied correctly at least based on my tests.

I've searched on the web for similar problems without success but finally found a string matching on delta walker changelog http://www.softpedia.com/progChangelog/Deltopia-DeltaWalker-Changelog-85293.html

"An alternate routine now attempts setting the Creation and Modified Date file attributes in cases when the destination file system, e.g. FAT32 accessible via samba, doesn't offer reliable implementation of these API."

So I realized the problem isn't in my code but somewhere in samba land (or Apple code??)

Now I've removed all Cocoa NSFileManager calls (also 'reading' not only 'writing' calls) and used only utimes() and I hope you can test again

The link to new alpha is http://cl.ly/1r1I0T0c0B3x

thanks in advance

Mr.Kiwi

2014-01-06 15:07

reporter   ~0000447

Ok, I test today the version 218-a3. - Result: better, but not correct.


Details:
I have on left side files/folders, like on screenshot. On right side, the folder (folder-2) is empty.

I start VisualDiffer from "terminal.app" like ./VisualDiffer.app/Contents/MacOS/VisualDiffer

I open an other tab (Cmd-T) with terminal.app (later for rsync calls)

I select all files in left side in VisualDiffer and choose "copy to right" action.
(All files will copy, after process is finished, VisualDiffer shown now diffs).

Immediately after VisualDiffer finished, in "terminal.app" I execute "rsync -avn folder-1/ folder-2"
--> I see no diffs, all is ok, only .DS_Store are different.

BUT, wait 10 Seconds, then the "terminal.app" tab shown an short "spinning wheel".

After them, execute "rsync -avn folder-1/ folder-2" again, and "BANG" timestamps are changed.

-----

Same behavior you can get, if you "copy to right" after this process is finished, press "refresh" on toolbar of VisualDiffer. - "Refresh" operation seems to flush data and modify timestamps.

-----

May be, you have an background process, that flush some cached operation? May be, "refresh" operations use NSFileManager?

-----

Additional, if I execute VisualDiffer on "terminal.app" now I get the log message:

CoreAnimation: warning, deleted thread with uncommitted CATransaction; set CA_DEBUG_TRANSACTIONS=1 in environment to log backtraces.

admin

2014-01-06 16:46

administrator   ~0000448

Last edited: 2014-01-06 16:49

View 2 revisions

> I start VisualDiffer from "terminal.app" like ./VisualDiffer.app/Contents/MacOS/VisualDiffer

Ok, I do the same steps, on left populated folder with subfolders (it contains the About Stacks.lpdf), on right empty folder


> BUT, wait 10 Seconds, then the "terminal.app" tab shown an short "spinning wheel".
> After them, execute "rsync -avn folder-1/ folder-2" again, and "BANG" timestamps are changed.

I can't reproduce this, the copy now preserves all timestamps and rsync doesn't report differences

> Same behavior you can get, if you "copy to right" after this process is finished, press "refresh" on toolbar of VisualDiffer.

This did happen with alpha2, now is fixed

> May be, you have an background process, that flush some cached operation? May be, "refresh" operations use NSFileManager?

There is a bkg thread but nothing is cached, I've checked before sending to you the alpha3

> CoreAnimation: warning, deleted thread with uncommitted CATransaction; set CA_DEBUG_TRANSACTIONS=1 in environment to log backtraces.

At this time you can ignore this warning

admin

2014-01-19 10:06

administrator   ~0000467

VD 1.5.8 is out but without a fix for this bug, I've a new alpha using a different approach so if you can help me again, you find the alpha 4 below.

On my test env, a winxp shared folder, it works.

thanks in advance for your patience

Please before use the alpha, update to v1.5.8 from mac app store


http://cl.ly/35423x1p2h30

Mr.Kiwi

2014-01-26 19:26

reporter   ~0000470

I have test your new version VisualDiffer-Bug218-a4.zip.
I use the same folder structure like last test.
Follow I see as result:

- Copy result have same content (md5sum check) - ok
- Copy/Sync result of files have same creation time and modification time like origin files - ok
- "Refresh" button after the "copy/sync" operation works well and no diffs are shown - ok
- Folders have different times stamps - ups
- Visual Differ ignores different timestamps for folders.

More details to folder timestamps:
- Visual differ show the timestamps correct.
- A new created folder (after copy/sync action) have the current timestamp.
- If I copy same with "rsync" the result folder himself have a timestamp like the origin, so the NAS can handle it well.

Conclusion:
- For files himself, the bug seem now to be fixed
- For folder timestamp, we can discuss, if this is a new bug issue.
- Please note, that I have read not all SMB implementations can handle folder timestamps well (depend on file systems). In my case, the NAS use XFS, that handle folder timestamps well, like the rsync test shown. But If you use FAT32, then the behavior of VisualDiffer (to ignore folder timestamps) may be correct.

admin

2014-01-29 17:46

administrator   ~0000471

Please apologize me for delay in reply.

The new folder modification timestamp isn't set by design not only on smb volumes.
The idea behind this decision was because folder timestamps are not used for comparison but now I realize this decision isn't correct so I'll update timestamp for new created folders, too.

I'm happy finally VD works on smb, I will publish v1.5.9 with this bug fixed very soon

Just if you want I can post here, when available, the version with the correct setting of new created folders timestamp.

Thanks again for your incredible useful feedback

admin

2014-02-05 17:45

administrator   ~0000476

The version at http://cl.ly/1k0a2d0n3V1x copies timestamps for newly created folders, just if you want to take a look at it.

If everything works I will publishing it in few days

Issue History

Date Modified Username Field Change
2013-12-30 21:14 Mr.Kiwi New Issue
2013-12-30 21:14 Mr.Kiwi Status new => assigned
2013-12-30 21:14 Mr.Kiwi Assigned To => admin
2013-12-31 08:11 admin Note Added: 0000424
2014-01-01 14:32 Mr.Kiwi Note Added: 0000425
2014-01-01 18:23 admin Note Added: 0000426
2014-01-02 07:55 admin Note Added: 0000427
2014-01-02 10:56 Mr.Kiwi Note Added: 0000428
2014-01-02 10:58 Mr.Kiwi Note Added: 0000429
2014-01-02 10:59 admin Note Added: 0000430
2014-01-02 11:23 admin Note Added: 0000431
2014-01-02 11:23 admin Severity minor => major
2014-01-02 11:23 admin Status assigned => confirmed
2014-01-02 11:25 Mr.Kiwi Note Added: 0000432
2014-01-02 11:36 admin Note Added: 0000433
2014-01-02 14:35 Mr.Kiwi Note Added: 0000434
2014-01-02 14:46 admin Note Added: 0000435
2014-01-02 15:06 admin Note Added: 0000436
2014-01-02 16:15 Mr.Kiwi Note Added: 0000437
2014-01-02 16:21 admin Note Added: 0000438
2014-01-02 16:22 admin Note Edited: 0000438 View Revisions
2014-01-03 13:15 admin Note Added: 0000439
2014-01-05 16:44 Mr.Kiwi File Added: screenshot-218-a2.png
2014-01-05 17:00 Mr.Kiwi Note Added: 0000440
2014-01-05 17:17 admin Note Added: 0000441
2014-01-06 13:55 admin Note Added: 0000446
2014-01-06 15:07 Mr.Kiwi Note Added: 0000447
2014-01-06 16:46 admin Note Added: 0000448
2014-01-06 16:49 admin Note Edited: 0000448 View Revisions
2014-01-19 10:06 admin Note Added: 0000467
2014-01-23 18:15 admin Summary Copy and Sync folders set invalid file time => Copy and Sync folders set invalid file time on samba volumes
2014-01-26 19:26 Mr.Kiwi Note Added: 0000470
2014-01-29 17:46 admin Note Added: 0000471
2014-02-05 17:45 admin Note Added: 0000476
2014-02-23 09:27 admin Status confirmed => closed
2014-02-23 09:27 admin Resolution open => fixed
2014-02-23 09:27 admin Fixed in Version => 1.5.9