How should we manage autoconfiguration files?

One of the features we’re working on for Thunderbird that I’ve mentioned before is the “autoconfiguration” setup, whereby when setting up an account, you type in your email address, and using the domain part (the part after the @ sign), Thunderbird tries to figure out what the right settings are. When it works, this is an amazing feature. This is not a new idea — in fact many mail programs have implemented in the past, including Exchange 2007 and the iPhone’s Mail program.

There are three ways of solving this problem: asking a central server for configuration files per domain, bundling some of those configuration files, and doing local network probing to look for likely domain names (mail.domain.com, imap.domain.com, etc.).

The central repository of configuration files has many advantages if we can figure out how to decentralize the provisioning of these configuration files, while remaining secure. In particular, it allows Thunderbird to “learn” about new domains or changed configurations as soon as those files are vetted, without having to wait for the next version of Thunderbird. Even with only a few configuration files, we should be able to cover the “short, fat, end” of the email domains — the top 200 domains for example probably cover 90% of home email users, which would be an amazing improvement for Thunderbird’s usability. (Gerv has already started a project months ago to collect some of this information)

Local port probing is most useful in corporate environments. Either their small user base or corporate policies may deter them from submitting their configuration files in the central server. In this case, the challenge is to strike the right balance of efficiency vs. completeness. We certainly don’t want to probe hundreds of ports on a local domain in a desperate search for a mail server, but to assume that all mail servers are configured the same way is unrealistic. A hybrid of the Exchange 2007 autodiscover model and the Mail.app configuration hostname-guessing model is likely optimal for this part of the market.

Side note: I could imagine an add-on that would bundle the configuration files and get updated periodically for those users who want to avoid leaking any data.

We welcome input on how we should manage these configuration files. The feature hasn’t landed yet, so it’s still very possible to influence this process. From my perspective there are a few key characteristics that should drive our process:

0) Reusing existing processes when possible makes life easier.

1) User experience is the goal. The point of this feature is to let as many people end up with a well-configured Thunderbird as easily as possible. This is why we’re going after a hybrid approach, trying to cover as many different environments with a single user experience.

2) This problem is not Thunderbird-specific. The data we’re looking for is not about how to configure Thunderbird, but how to configure IMAP/POP/SMTP email clients in general. If Mozilla can help solve the problem of email configuration in a safe, scalable, trusted way, then I’m fine with other email clients using it. I see this as potentially a very powerful public service that Mozilla can provide to email users everywhere.

3) We want to decentralize contributions of configuration files, but control the review processes, to facilitate end-user contributions and cover the long tail of domains, with a consistent policy especially with respect to validation.

In the short term, it feels like we could easily start with a variation on the processes we already use for code, with a security review overlay. In particular, I would suggest as an initial draft that:

– we have something equivalent to a module, which contains the published configuration files and processes around them
– each configuration is reviewed by someone other than the configuration submitter (patch author)
– each configuration needs to be validated by the reviewer against public documents published by the mail provider (ISP, university, etc.)
– Mail administrators can file a bug, or contact the module owner directly if they want to report changes in their email setup or suggested changes.

We will probably want to figure out more sophisticated systems involving cooperation with mail administrators (using MX or DNS records, or signed emails, or …) in a later phase, but I don’t think we should gate being able to serve users of the largest domains (gmail, yahoo, major ISPs, universities). Walk, then run, should be our approach.

Similarly, there are possible futures where users could share configuration files with each other, but I don’t think we’re ready to tackle those securely yet.

We’ll figure out a way for those interested in submitting or reviewing configurations to sign up in a later round. At this point, we’re most keen on process-type feedback. Comment on the bug, or here, thanks!

PS: Whatever process we arrive at, we’ll then apply to the files we’ve been using in testing and the configuration data in Gerv’s spreadsheet, to make sure that all the data has gone through the right hoops.

26 Comments

  1. While Apple’s Mail definitely includes some per-domain configuration details for some domains, it seems like it’s also running a few commands on the server to try to figure out its structure. It has figured out all of my domains, for instance, despite them having different structures and me being the only user of them.

    Like

  2. 1. There’s a webpage with an HTML form somewhere, where anybody can
    enter the info for ISPs. That includes the detailled settings, but also the webpage with the ISP’s instructions. That goes into a “staging” database.
    2. We (MoMo) run some automated tests on that data, like checking
    that the server hostname domains and email address domains match
    and maybe even have a script which tests the server capabilities
    (like our guessConfig.js, or exactly that). This is a first pass
    verification.
    3. We can have a manual verification as well, either only for these
    datasets which fail the above tests, or for all.
    4. Preferably, there should be somebody with an account at the
    relevant ISP who can confirm that these settings work. Preferably,
    this person is also trusted by us. Localizers are a good contact
    group here maybe.
    5. Once we are confident about the settings, they go into a
    production DB. Preferably, that is versioned.
    6. From the production DB, the XML files are generated (probably as
    static files).
    7. The static files are served by an Apache.

    Like

  3. Also, as described on the project page, there’s a step which tries to fetch the config from the ISP, not a central store. This is useful for intranets and the “long tail” (many smaller sites).

    The central database is mostly for the top 200 mail providers, which should cover 90% of non-corporate accounts, as David wrote.

    Thanks a lot, David, for the link to the MS spec! I vaguely knew that something like that exists, but had no details. It looks extremely similar to the ISP fetch that I designed and implemented. I’d like to implement Microsoft’s system as well – then we can cover the Exchange server, which (unfortunately!🙂 ) covers a lot of the corporate users, without having to guess.

    Like

  4. My suggestion is to use DNS SRV, like Jabber uses to separate servers and clients.

    For example, to find the server for the user at user@example.com to use, the client would look up _imap[s]-client._tcp.example.com. IN SRV.

    The authentication capabilities should be negotiated upon connection to the designated server. One benefit of this method is that it allows for multiple servers to be specified.

    I would actually like to see DNS SRV replace MX for server-server communication as well (otherwise you could argue, why not just keep adding record types for other well-known services – FTP, WWW, etc).

    Like

  5. How about Mail providers of each locales?
    Too XXX mail providers will be consist mainly of en-US and some large locale/country.
    We mozilla should provide same experience not only for en-US but also for every locales. For non en-US locales, we should collect main Mail provider information for each locales, not only large country.
    How about the plan about autoconfig support for each locles? I think small l10n communities cannot collect enough Mail server information of their locale (since they cannot take time to talk with ISPs) and we Mozilla should support collecting/contacting ISPs of such locales.

    Like

  6. Dynamis: We definitely want to decentralize the data collection, as no one can know better than people in smaller locales which mail providers are important. It doesn’t have to be localizers if they’re busy, it can be other kinds of locale-aware contributors.

    Raffael, Chris: there are lots of arguments for and against MX, DNS, DHCP, etc. As I said, I don’t think now is the time to solve that problem, until we know that the rest of the system works. In particular, I think different corporate environments work very differently, and I think we need to diffuse this idea out with the first release of the feature and see what feedback we get.

    Like

  7. Beside auto configuration (smtp/pop/imap), it would be extremely useful for corporate environments a policy enforcing feature. Thunderbird should check for a specific document via http GET (document URL should be configured manually, of course, like proxy auto configuration for web browsers).
    The policy document could be an xml document with a “global” container with settings common to every account and optional “account” (email) containers with options specific to a single account.
    Options could be in form optionvalue.

    My two cents.

    Like

  8. You already have the answer
    (1) ship the Thunderbird install with an “email-auto-discover” plugin
    (2) during installation, check that plugin for updates (which may happen already)
    (3) update that plugin whenever you have new rules/mail domains/whatever

    Like

  9. I am with Chris Hills. Using SRV records is something which can solve the problem without requiring maintenance and administration tasks from the Mozilla side (DB servers, setting editors, dealing with authenticating mail provider stuff members to update their records etc.).

    I don’t know any common use for SRV in the public (no, Jabber and SIP are not common enough yet) and this can make SRV records more useful and keep the lists maintainable by authorized personals only (i.e., people with access to the domain records).

    Since for most people setting up the mailbox is a one-time task, I don’t see any real use for a client-side providers list or anything similar to that.

    Like

  10. From my past experience working in very big corporate environment, I think you can put the corporate world apart as the way it works (at least in France ) is :
    1) IT dept. decides to deploy the solution
    2) IT dept. packages the solution.
    3) Packages is tested, if NOK goto 2 else goto 4
    4) deployed to a sample of user, feedback is gathered goto 2 or 5 depending on feedback
    5) deployed to entire staff.

    What needs to happen for those corporate user is a very easy way to make the configuration themselves and a mechanism that will take the configuration when the deployment/installation is done. Idea’s here is to document the format of the conf. file or to provide a wizard to create the file.

    How about having templates too, something for hosted domains on google, yahoo, dreamhost etc… Like I have the choice between a google autoconfiguration file that will force my email to @gmail.com or googlemail.com for uk resident. And another “template” where I use the services from google but with emails being @hirlimann.net

    Like

  11. In corporate environments the IT department will configure the user’s desktop. With this in mind, it would be useful for IT support to have a tool that would configure a startup script that when used would run TB with the user’s correct configuration. This script might be placed in the user’s home directory or dynamically created via a web page. In general, help IT and don’t get in their way.

    For non-corporate environments get the MX record for the email domain and, as you said, see which of the top dozen ISPs handles the email and use a centrally administered database.

    Like

  12. Just make sure that the setup and the how-it-works of the centralized server you’re using for now is openly documented as well as the client-side pointer to it is configurable, so that nobody needs to feel like this is a lock-in to a specific centralized service, then I think this is good enough for the first step. I also think though that a generalized way of doing this, probably through some DNS-based or -triggered approach is the best idea for the long term.

    Like

  13. I’m working at an ISP with a number of users. It doesn’t qualify as top 200, though. As we insist that users use secure connections to connect to our mail servers, configuration of mail clients is a major support issue for us. Mail clients still are not assuming that secure mail connections are the default. That is a big problem, because it costs us a lot of money (in support time) to keep our users safe.

    From this kind of view we would really appreciate a setup where we can lower the support burden by providing some kind of auto configuration file to the user. As we are far from top 200 though, I doubt that you would consider including our auto configuration file in your central database. We would therefore appreciate, if we could simply set up, say, a txt record in DNS for each domain name with a URL pointing to an XML file with the auto configuration data. As long as you trust DNS (which might be an issue as well) this scenario is also relatively safe for the user. It would be in our interest to provide such a file as it would safe our users time and us a lot of money. If this would be an industry standard – even better for us.

    So don’t count on your users alone. ISPs are also interested in this auto configuration thing as it has the potential to lower support time considerably.

    Like

  14. Please see the project page, many of the proposals have already been discussed in the newsgroup discussion linked there.

    Problem with DNS is: 1) insecure (easy to forge) 2) hostname, port is just not enough. We cannot discover all necessary information from the server itself either, e.g. IMAP root folder or email address -> username mapping, which is not standardized and many people have serious problems with.

    We already give the ISP/company a way to provide us with a config.

    But there’s also a central database for those many ISPs which do not cooperate initially. David A. was asking how we can manage a central database.

    Like

  15. Patrick: I don’t think there’s any reason we wouldn’t host your ISP’s config file, no matter where you are on the long tail. DNS based approaches might work as well.

    Like

  16. How about something similar to ‘Google Site Map’, e.g. ‘Mozilla email settings’. It could be a xml file hosted by the ISP/mail account provider stating all information needed to set up the accounts.

    I’m aware that this would need the ISPs to take this one on but why not… It would enable any client to use the settings.

    Like

  17. I’d like to see a format that can be supported by any ISP effectively. I don’t think it would be an improvement if only the top five mail providers receive an easy solution – this we have already IIRC.

    Implementation of such a protocol should be fairly easy and serve 98% of all mail providers equally. This was the reason for my proposal at https://wiki.mozilla.org/Thunderbird:Autoconfiguration:DNSBasedLookup by mimicking SRV records of XMPP somewhat.

    Like

  18. Concerning security of the DNS approach: It is equally secure as mail itself. MX records are served by DNS and a compromised DNS server would have the same effect in both instances. It’s currently more likely that the “mailconf” file is served from a secured location (HTTPS) than mail servers actively supporting SSL. There isn’t anything specially new with DNS discovery and if implemented correctly reasonable secure (for settings up mail accounts).

    Additionally I wouldn’t like to depend on a centrally managed DB by Mozilla for various reasons. I think it important to be able to change the “mailconf” at will without the need for a third party.

    Like

  19. Why not make it part of AMO?

    On the client side (Thunderbird itself) people could search for their ISP’s/Service’s name and optionally select their country from a list.

    As Ben proposed (comment 2), anybody could add data for it. Alternatively, groups would be formed to work on (global) coverage of mail settings, maybe like translations are handled?

    Also, doesn’t wikipedia have lists of ISPs per country?

    Like

  20. Hi David,

    what do You think about a completely decentral approach?
    The user enters his eMail address only, let’s say george@gmail.com, the autoconfig plugin transformes it in a defined way into an URL e. g. http://gmail.com/mailautoconfig?user=george or something like that. This should return an XML file in a defined form which describes how to set up SMTP/POP/IMAP. With this information the autoconfig plugin could do this automatically.

    (It’s something like the proxy autoconfiguration.) With this approach every admin has the possibility to provide autoconfig information for his own customers, he can change them when ever he wants to change, and the authentication of the publisher is impicitly given.
    Another advantage is, that also many little providers can offer those autoconfig data without having one huge central repository.

    For reasons of authentication maybe we could also think about https-URLs.

    Maybe in a second step You could ask a central repository, if in the first step an Error 404 (not found) was returned. But in this case I would suggest to ask the user first.

    Hope it helps.

    Best regard,
    Bernd Stroessenreuther

    Like

  21. To support cases where many domains are served by the same mail servers (such as Google Apps, or other hosted services) you could easily look up MX records now, without any action on administrators’ part. There’s no possible way to know all the domains using Google Apps, but if you look up a domain and its MX is google.com or googlemail.com then the rest should be straightforward…

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s