3

We have a site deployed in Norway, and people have uploaded PDF-assets with Norwegian non-ascii characters in the filenames.

Normally this works fine, the URLs are encoded like this and it works:

http://[oursite].no/assets/files/downloads/%C3%85rsberetning-2014_kladd.pdf

("Årsberetning" - annual report)

But one file didn't want to download, it only gave me 404. However, checking on the server the file was there, and the filename appeared correct. Also, re-scanning asset folders didn't remove it.

BUT - I was not able to rename the file from within craft, nor download it.

After much hair-pulling I established that the file on disk had the Å encoded as '\x41\xcc\x8a' whereas craft looked for a "pre-composed" Å ('\xc3\x85').

NOW the question - where did this break? :)

Does Craft unicode-normalise filenames? Does MySQL normalise the unicode it stores?

The file has moved around various developer machines and staging servers before making it into production, it could be some other part of the tool-chain that broke it.

Improve this question
2
  • I'm having problems with this all the time with norwegian characters. For me, the problem normally happends when moving files between mac osx and various linux servers. As long as the files is uploaded to craft to a server, and stay on that server, it works fine.
    – André Elvan
    Commented Apr 16, 2015 at 18:55
  • André - yeah, that is my experience as well. On our linux staging server the file was uploaded, there it worked fine (but the file was stored twice, with both encodings...) - then it went through git onto a osx machine, and from there it was pushed live with rsync. It's actually a miracle that anything ever works at all :)
    – gromgull
    Commented Apr 17, 2015 at 8:38

2 Answers 2

Reset to default
3

I think this is going to be one of those impossible to answer questions because of so many unknown variables.

MySQL could be screwing it up because it depends on 1) MySQL server configuration and 2) The charset and collation of the database, the tables, the connection strings, etc.

The file system itself could be screwing it up because they all tend to have different treatments for character encodings (HFS, NTFS, FAT, EXT, etc.).

Throw in the fact that you said this has been on various developer machines and touched by an unknown number of other programs, it makes it very difficult to diagnose.

Improve this answer
5
  • 1
    Given this, and I've requested this several times both for the EE and Craft versions of Craft, can we please, please, please have a setting to convert filenames to ascii, like slugs? :)
    – André Elvan
    Commented Apr 16, 2015 at 18:57
  • @AndréElvan Already on our internal list. :)
    – Brad Bell ♦
    Commented Apr 16, 2015 at 19:02
  • Brad: I totally understand :) I just wanted to collect experiences from other people!
    – gromgull
    Commented Apr 17, 2015 at 8:36
  • I also take this to mean that "Craft does not intentionally apply unicode normalisation to filenames (or to anything)" - so it happened "somewhere else".
    – gromgull
    Commented Apr 17, 2015 at 8:38
  • 1
    @gromgull Correct... Craft isn't doing any filename normalization currently.
    – Brad Bell ♦
    Commented Apr 17, 2015 at 15:57
4

This has now been solved by the addition of the convertFilenamesToAscii config setting introduced in Craft 2.4.

If you are working on a multilingual site, you might also find the limitAutoSlugsToAscii config setting useful.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.