Wednesday, May 23, 2012

Trying to fix line-endings with git filter-branch, but having no luck


I have been bitten by the windows/linux line-ending issue with git. It seems, via github, msysgit, and other sources, that the best solution is to have your local repos set to use linux-style line endings, but set core.autocrlf to true. Unfortunately, I didn't do this early enough, so now every time I pull changes the line endings are borked.



I thought I had found an answer here but I can't get it to work for me. My linux command line knowledge is limited at best, so i am not even sure what the "xargs fromdos" line does in his script. I keep getting messages about no such file or directory existing, and when I manage to point it to an existing directory, it tells me I don't have permissions.



I've tried this with msysgit on windows and via the Mac OS X terminal. Any help would be GREATLY appreciated.


Source: Tips4all

5 comments:

  1. The easiest way to fix this is to make one commit that fixes all the line endings. Assuming that you don't have any modified files, then you can do this as follows.

    # From the root of your repository remove everything from the index
    git rm --cached -r .

    # Change the autocrlf setting of the repository (you may want
    # to use true on windows):
    git config core.autocrlf input

    # Re-add all the deleted files to the index
    # (You should get lots of messages like:
    # warning: CRLF will be replaced by LF in <file>.)
    git diff --cached --name-only -z | xargs -0 git add

    # Commit
    git commit -m "Fixed crlf issue"

    # If you're doing this on a Unix/Mac OSX clone then optionally remove
    # the working tree and re-check everything out with the correct line endings.
    git ls-files -z | xargs -0 rm
    git checkout .

    ReplyDelete
  2. The git documentation for gitattributes now documents another approach for "fixing" or normalizing all the line endings in your project. Here's the gist of it:

    $ echo "* text=auto" >>.gitattributes
    $ rm .git/index # Remove the index to force git to
    $ git reset # re-scan the working directory
    $ git status # Show files that will be normalized
    $ git add -u
    $ git add .gitattributes
    $ git commit -m "Introduce end-of-line normalization"



    If any files that should not be
    normalized show up in git status,
    unset their text attribute before
    running git add -u.

    manual.pdf -text

    Conversely, text files that git does
    not detect can have normalization
    enabled manually.

    weirdchars.txt text

    ReplyDelete
  3. The "| xargs fromdos" reads from standard input (the files find finds) and uses it as arguments for the command fromdos, which converts the line endings. (Is fromdos standard in those enviroments? I'm used to dos2unix). Note that you can avoid using xargs (especially useful if you have enough files that the argument list is too long for xargs):

    find <path, tests...> -exec fromdos '{}' \;


    or

    find <path, tests...> | while read file; do fromdos $file; done


    I'm not totally sure about your error messages. I successfully tested this method. What program is producing each? What files/directories do you not have permissions for? However, here's a stab at guessing what your it might be:

    One easy way to get a 'file not found' error for the script is by using a relative path - use an absolute one. Similarly you could get a permissions error if you haven't made your script executable (chmod +x).

    Add comments and I'll try and help you work it out!

    ReplyDelete
  4. okay... under cygwin we don't have fromdos easily available, and that awk substeb blows up in your face if you have any spaces in paths to modified files (which we had), so I had to do that somewhat differently:

    git status --short | grep "^ *M" | sed 's/^ *M//' | xargs -n 1 dos2unix


    kudos to @lloyd for the bulk of this solution

    ReplyDelete
  5. git status --short|grep "^ *M"|awk '{print $2}'|xargs fromdos

    ReplyDelete