Students! Teachers! Lend us your ears!

Here at Mapbox, we value the creative ways our tools are being used in the classroom. That’s why last week we launched Mapbox Education and made our tools totally free for students and teachers.

We want you to use Mapbox as your sandbox for learning, creating, and growing. We have a ton of tools you can put together to build beautiful, smart, interactive mapping projects. With 1GB of storage, 3 custom map styles, and an ever-growing range of guides, examples, and documentation, students can jump right in while instructors can design an updated curriculum. Plus, all our tools are open source and can run on any platform – Mac users rejoice!

The world of geospatial technology is constantly changing and growing, and at Mapbox we’re running right along side it. Sign up for Mapbox Education today and stay ahead of the curve.

Questions? You can contact me on Twitter at @lyzidiamond or via email at education@mapbox.com.

FOSS4G-NA 2015 starts today in Burlingame, California, and I could not be more excited. These conferences are such an excellent way to follow the great projects our peers are working on, glean information from what they’ve learned, and share our own experiences working in open source geospatial. And with so many interesting talks, it’s sure to be a busy week.

Image from Maptime DC

But if it’s your first time at a geo conference, or even your first time at FOSS4G, it’s easy to feel overwhelmed. There are so many talks and so many people: it’s hard to know where to start!

Tomorrow morning, I’ll be giving a talk at FOSS4G-NA called On-Ramps to Open Source, 11:15 a.m. in Regency A. My goal is to help you make heads and tails of the open source geospatial landscape; there are a lot of roads you can go down to get to the open mapping highway, and I want to give you some directions. By the end of this talk, you should have a better understanding of what tools you can use for certain projects and how to get started using those tools.

This talk coincides with the recent launch of Mapbox Education, our effort to make Mapbox totally free for students and teachers. We want everyone to have a chance to experiment with our tools, and Mapbox Education is a huge stride in that direction. If you’re interested in helping make it great, come to my Mapbox Education Birds of a Feather session on Wednesday at 7:30 p.m. in Regency C. If you’re teaching with Mapbox tools, or would like to, come share your experiences. What works and what doesn’t? What resources can we provide to support you?

What would a geo conference be without an appearance from Maptime? On Thursday at 1:45 p.m. in Regency A, come see me and Beth Schechter from Stamen talk about this open source community movement in What Time Is It? It’s Maptime!. Our talk will focus on what we’ve discovered through facilitating an open-source geospatial learning community, and we’ll give guidance on starting a Maptime or other educational project in your city.

As more and more people join the mapping family, we become stronger, grow faster, and have the opportunity to learn so much from each other. This week at FOSS4G-NA 2015, let’s build our skills, foster some new friendships, and make our community more welcoming for newcomers. If you want to chat Mapbox Education or really anything this week, hit me up at @lyzidiamond or lyzi@mapbox.com.

This year we’re supporting 10 scholars to come to State of the Map US, what will be the largest open mapping conference - and OpenStreetMap conference - of the year. OpenStreetMap is made up of a community of people from all over the world and many different backgrounds, working together to make the best map of the world and keep it open for all to use. Our conference should represent that, and scholarships help bring people who are new to the community or may not have the backing of a company to pay for the trip.

If you’re doing something interesting with mapping, open data, or OpenStreetMap, we encourage you to apply for a scholarship. The application is short, and the deadline is coming up this Sunday, March 15. Also, consider proposing a talk.

You can sponsor scholars too! You can sponsor a single sponsor when you buy your own ticket, or hit up cathryn@openstreetmap.us about supporting many.

One of the most challenging geographic data sets I have ever worked with is theNew York taxi trip origins and destinations that Chris Whong obtained through the New York Freedom of Information Law. It is huge (375 million points), tightly concentrated (nearly all within a few miles of Manhattan) and varied (with some streets only visited once and others more than a million times).

My previous map of this data was a set of static bitmap tiles rendered withDatamaps, where I could handle the wide range of densities by drawing all the dots, accumulating brightness linearly, and then taking the cube root of the accumulation to condense that wide range into the available range of pixel values. That’s hard to do with vector tiles because tiles that contain more than about 200,000 points are too big and too slow to render on demand. So high-density data has meant giving up the resolution-independence and scalability of vector tiles.

You no longer need to make that tradeoff because Tippecanoe can now selectively drop points in the densest areas to moderate their size and visual impact while retaining full detail in sparser areas. The vector tiles forthe map below were made with

tippecanoe -z19 -d11 -g3 -o nyctaxi.mbtiles

for a base zoom level of 19, a tile resolution of 2048 (2¹¹) at that zoom, and a gamma reduction of 3, which, at each zoom level, reduces the density in areas where points are less than one pixel apart to the cube root of their original density.

How it works

Reducing density by counting points within bins doesn’t look very good, and k-means clusters or voronoi diagrams are too slow to construct for such a large number of points. What worked well was to compute the z-order index of each point, the same index that is used to divide maps into tiles, and manipulate the density along that linearization of the point locations instead of trying to do it in space. At zoom level n, the indices for points that are about a pixel apart on screen will tend to be about 2^64-2*(n+8) apart, and the places where this is not true, where indices jump around at tile boundaries, will average out over a large number of points.

Even along a line, you don’t have a continuous density function, you have a series of discrete points, but instead of thinking about the density of the points, you can think about the sparseness of the gaps between successive points. The algorithm goes through the points in index order, and at each point looks at the size of the gap between its index and the previous point’s index. If the difference is larger than “a pixel,” as defined above, the gap is sparse enough, so it copies that point into the tile and goes on to the next one. If the gap is too small, it needs to create a gap whose size (relative to the pixel size) is the cube of the gap that it found. So it skips over the current point and over as many more as it needs to until it gets to one that is far enough from the previous point that was added to the tile. With a sufficiently large gap created, it copies that point into the tile and goes on to check the size of the next gap.

The cube root gamma for the taxi map has nothing to do with the eye’scube root perception of lightness, which monitor gamma compensates for. It just looked about right and reduced the tile density enough that it would work, cutting the million points in the zoom level 19 tile at Penn Station to 150,000. With data whose densities are less extreme, something closer to a square root seems more appropriate. Automatically choosing the optimal gamma for a given data set is a research project for the future.

Given a GPS trace, how do you find the matching streets in a given road network? For instance, think of the GPS trace of a taxi making its way through a city. How do you identify the actual streets the taxi used? Solving this problem unlocks important applications like improving street network data or computing traffic statistics.

The challenge is that GPS is prone to disturbances and inaccuracies. Any given position in a GPS trace may be off by a few feet or a lot more. Consider two one-way streets going in opposite directions among tall buildings. The buildings introduce a fair amount of noise.

The naive approach would be to chose the nearest street segment for each GPS point resulting in the red line. Clearly, this is not desirable, especially since the street could not even be traversed in that direction! Luckily there’s a solution to this problem introduced by Paul Newson and John Krumm in their 2009 paper on map matching.

A key component of Newson and Krumm’s approach are so called Hidden-Markov-Models which are based on Markov Chains. Markov Chains are popular as many random processes can be modeled with them. Given a set of states and a function that outputs the probability of transitioning from one state to another state it yields a mathematical model of the process. A typical question a Markov-Chain helps answer is: Given state A, how probable is state B after 2 steps?

Lets say the states of our Markov Chain are street segments. How would you measure the probability to transition from one street segment to any other street segment? It is fair to assume that a taxi can only transition to streets that are nearby. Basing probability on driving distance seems like a good idea. However the taxi might drive faster, which would change the distance between the sample points and ultimately the transition probabilities. Newson and Krumm found that a large difference between the driving distance of two street segments and the straight line distance between the corresponding GPS samples point to an unlikely transition. More precisely the difference of the two values follows an exponential distribution. That distribution can be used as model to estimate the transition probability.

This equips us with the tools to measure the probability of a given sequence of street segments. However we want to find the most probable sequence of street segments for given trace of GPS points. This is where Hidden-Markov-Models come in. They where popularized and applied by Lawrence Rabiner in his paper from 1989 on speech recognition. Hidden-Markov-Models enable you to find the most probable state sequence for a given sequence of observations.

For map matching, the observations are the GPS points. For each street segment the probability of a given GPS point matching the street is inverse-propositional to its distance. More precisely, Newson and Krumm model this using a Gaussian distribution:

Once we are able to compute the probabilities of observation and transitions we need an algorithm to identify the most probable sequence of states. This problem can be solved with the Viterbi Algorithm. The basic idea is to chose the most probable previous state as the predecessor of each state in a sequence. This way the algorithm extracts the entire state sequence by going back from the most probable last state.

This map matching approach is now available 100% open source in form of a plugin for the Open Source Routing Machine which we just merged back into the mainline branch. To learn more, join us tomorrow in our OSRM routing session at FOSS4GNA led up by my colleague Dennis.

Processing pipelines for large geospatial datasets is a big topic here at FOSS4GNA this year so we’re excited to share our work in two talks today.

Wrangling pixels at FOSS4G

Wednesday March 11, 5-6PM Regency B

This evening at FOSS4G, Charlie and I will be talking about how we build a cohesive global basemap from a variety of imagery sources. We will give insight into the most important open source tools in our workflow, how we approach some of the strange imagery processing quirks that come up, and how we scale our methods into a large volume pipeline.

This hour will be filled with lots of beautiful satellite and aerial images in their pre- and post-processed states, visual examples of weird issues we encounter, and code snipets to help you get started with open source raster processing.

Geoprocessing with Node.js at FOSS4G

Wednesday March 11, 5-6PM Regency A

Carol, also speaking tonight, is excited to be giving a talk on using Node.js to explore and manipulate geodata. She’ll be sharing how she uses Node in the context of maps to process and render huge amounts of data, and she will provide examples of how to use libraries like node-mapnik and node-gdal. This talk will offer an approachable look at javascript as a geoprocessing alternative and as a doorway into both the geo and web development worlds.

We are all really looking forward to sharing with you, and can’t wait to see you tonight!

See the full schedule of talks over at the FOSS4G NA web site and all Mapbox talks here on our blog.

Today is the final day of FOSS4G 2015. In the morning session we will share highlights of our work on Mapbox GL. Our talks will focus on the open source software, design, and formats behind our vision for the future of dynamic web mapping.

Leaflet, WebGL and the Future of Web Mapping

Vlad will speak about Leaflet, Mapbox GL, and the pursuit of simplicity in software.

Thursday March 12, 9 a.m. Regency B

Cartographic design in GL maps: beyond zoom levels and raster tiles

Nicki will discuss the new cartographic capabilities and paradigms for styling in Mapbox GL.

Thursday March 12, 10:30 a.m. Regency B

State of Vector Tiles

Dane will speak about the motivations, history, and future of the vector tile specification that powers Mapbox GL.

Thursday March 12, 12 p.m. Regency B

In this example, Turf is processing and visualizing the largest historical point dataset on Socrata and data.gov. The interface below visualizes every 311 call in NYC since 2010 from a dataset that weighs in at nearly 8 GB.

The bottom slider is draggable, so you can loop over a specific time segment. It uses D3’s steamgraph feature to present 311 totals per borough across time. Lee Byron and Martin Wattenburg originally described this technique as it was used in the New York Times, saying of steamgraphs:

[…] the twin goals are to show many individual time series, while also conveying their sum. Since the heights of the individual layers add up to the height of the overall graph, it is possible to satisfy both goals at once.

The preprocessing step is very similar to our recent post on Chicago historical crime data. The code uses JSONStream to scale the aggregation algorithm beyond Node.js’s default RAM limits with piped data and incremental computation. Unlike a traditional GIS database, Turf’s flexibility allows for composable algorithms that scale well past what fits into memory or even on a single machine.

Want to see more examples like this one? Let us know @mapbox and check out our starter guide to Turf analysis.

With more than 50,000 trails, AllTrails.com guides you to the perfect spot to hike, bike, ski, snowboard, and camp in the backcountry. One million people use the AllTrails app every month to plan their outings and cache maps on their phones so they're available offline.

The AllTrails team was able to design a custom map that both enhances terrain and elevation details while incorporating their extensive global and national datasets into one custom vector tile data source for styling.

Alltrails custom data layers in Mapbox Studio source mode.

Seamless data and design

AllTrail's custom data includes national peaks, elevations, trail intersections, distances and global park boundaries. From regional overviews to ground-level trail details, data and design were fused into one outdoors map using Mapbox Studio, our map design studio. The final product is a highly functional map for mobile and offline usage.

Check out the app or visit AllTrails online

Perfect for planning your next big excursion, you can download the AllTrails app free from the iTunes store or at AllTrails.com. AllTrails Pro members can start printing and planning with the new map design.

The Mapbox Vector Tile spec is catching on. We are pleased to announce that Esri is the latest company to embrace this open format for fast and efficient distribution of tiled geodata.

Esri made the announcement last week at their annual developer conference as they revealed that they are building a mapping framework on top of vector tiles.

According to several Esri devs, they plan to contribute improvements to the specification on github.com/Esri, which is exciting. Open specifications describe open formats, and open formats are the backbone of open data.

(Lego pic courtesy of Martin Copping)

OpenStreetMap allows any map feature to have a name for labeling points, streets and areas on a map. These names can be in any language by use of multilingual name tags.

Name tags for the major world languages - Deutsche, English, Español, Français and similarly local languages can be rendered in Mapbox Studio with simple changes in the CartoCSS editor and the OpenStreetMap derived mapbox-streets datasource.

To render other languages not available in mapbox-streets, a few extra steps are necessary to export the relevant GeoJSON and multilingual tags from OpenStreetMap using the Overpass Turbo API and creating an additional vector data source in your Studio project.

At the Bengaluru Mapping party last week, we created the first street map of Bengaluru in Kannada. Yogesh, who has been translating the street names documented the process here.

If you enjoy working on map data and open source technologies, take a look at our open positions with the data team at Mapbox Bengaluru!

Emily McAfee is joining Mapbox! Emily graduated from General Assembly in Washington, DC where she studied web development. She is diving in on our Upload and Data APIs, working to making them powerful and developer-friendly. She’ll also be developing on our data processing pipelines to make sure we can process any type of data and publish it quickly.

Follow Emily on GitHub and Twitter!

Today we’re releasing the first edition of Landsat-live, a map that is constantly refreshed with the latest satellite imagery from NASA’s Landsat 8 satellite. Landsat 8 data is now publicly available on Amazon S3 via the new Landsat on AWS Public Data Set, making our live pipeline possible. We’re ingesting the data directly from Amazon S3, which is how we’re able to go from satellite to Mapbox map faster than ever. With every pixel captured within the past 32 days, Landsat-live features the freshest imagery possible around the entire planet.

Landsat live: a live updating imagery pipeline of the world at 30 meter resolution. Click the magnifying glass to search and jump to a new location. The map is continuing to fill in live. Areas that were fully clouded on recent satellite passes don’t have data – check in again soon.

With a 30 meter resolution, a 16 day revisit rate, and 10 multispectral bands, this imagery can be used to check the health of agricultural fields, the latest update on a natural disaster, or the progression of deforestation. Interact with the map above to see the freshest imagery anywhere in the world. Be sure to check back often and observe the constantly changing nature of our planet as same day imagery hits this constantly updating map. Scroll down the page to see some of our favorite stills of the earth from Landsat’s latest collection.

A huge thanks to our friends at AWS for hosting this imagery in their cloud infrastructure and to Frank Warmerdam at Planet Labs for contributing open source software to transfer the data to S3.

The city of Antalya, Turkey is bordered to the north by the Taurus Mountains and to the south by the Mediterranean Sea. Antalya is reaching the end of its cooler season. As the hot, dry season sets in, vegetation change will be able to be seen from this view. Captured March 16, 2015.

Brukkaros Mountain is an ancient volcano in southern Namibia. Brukkaros reaches up 650 meters tall, surrounded by very flat landscape. To the far west, where the landscape changes again, old river beds that have become dry canyons can be seen. Captured March 8, 2015.

The town of Denison in western Iowa is still in the midst of winter weather this week. A dusting of snow can be seen across the agricultural fields that will be green and vibrant with crops come summertime. Captured March 14, 2015.

Marrakech, Morocco sits close to the equator and sees warm temperatures the year round. Palm trees and olive orchards make up a large amount of the vegetation around this city, which will change subtly as they bloom throughout the year. Captured March 9, 2015.

The east side of Lake Michigan shows signs of a long, cold winter with massive ice sheets having frozen nearly halfway across the width of the lake. Snow can be seen covering the rural eastern side of Lake Michigan. Chicago, to the west, is a more subtle grey due to the footprints of buildings and plowed streets. Captured March 11, 2015

The extremely fertile Nile River Delta in northern Egypt is a rare green gem surrounded by desert landscape. Come autumn time, blooming lotus flowers will add a slight but discernible shade of red to the thick vegetation. Captured March 18, 2015.

The Himalaya Mountains in their natural snowy state will change slightly throughout the year. The topmost parts of the mountains will remain covered with ice and snow, but the snow line where bare rock and arctic vegetation start to creep in will rise and fall with the seasons. Captured March 4, 2015.

The Central Valley in California is an active, year-round food production epicenter. While green fields can be seen down in the valley throughout the year, it is the hills that border the valley on both the east and west sides that will turn brown as the summer season approaches. Captured March 7, 2015.

We’re just getting started. If you would like to use this imagery via your Mapbox account, send us an e-mail at satellite@mapbox.com to get an early preview.

This Monday, join us for “Designing and Publishing a Map Product with Mapbox” at PARISOMA in SF! In this 90-minute workshop, we will provide a concise and comprehensive look into building maps with Mapbox. We’ll start by introducing some Mapbox tools, such as Mapbox Studio and the Mapbox.js API, then dive in and construct a map with real-world data.

This workshop is best suited for folks who have a little HTML, CSS, and JavaScript under their belts, even if it’s just the tiniest bit. If you’re interested in joining us, RSVP here before Monday — you can use the code Mapbox15 at checkout to get a 15% discount.

If you have any questions, feel free to hit me up on Twitter at @lyzidiamond or at lyzi@mapbox.com.

Eliane Joyo Meneses is joining the Mapbox team! As part of the data team, Eliane will focus on improving OpenStreetMap based on customer feedback and strategic data needs. Eliane joins us from DIRESA where she helped implement inventory software. Follow Eliane on GitHub and Twitter and watch out for her work on OpenStreetMap.

Today we updated our privacy policy and law enforcement guidelines to clarify that we’ll never disclose user location information to law enforcement officials except in response to a probable cause search warrant, or in case of a life-threatening or similarly dire emergency. This guarantees the strongest possible legal protection for location data, equal to the guarantees we’ve already made about other user content. Here’s the new language on mapbox.com/privacy:

We will only disclose user content, including maps or data, or location information, in response to a probable cause search warrant. If we have a good faith belief that there is an emergency involving the danger of death or severe physical injury, we may also provide the limited information necessary to prevent that harm, if we have it.

Location data is highly sensitive: it contains information about where we live, our daily habits, and our network of friends and acquaintances. As we look into increased use of probe data to help us produce the world’s best maps and routing, we’re paying close attention to the privacy of this information. Technical protections and robust anonymization form the backbone of our work, but legal protections are essential as well.

Here’s why the new legal language is critical: existing U.S. statutes let law enforcement officials access certain types of electronic records under reduced legal standards. Disclosure orders can be obtained for any information “relevant and material” to an ongoing investigation. For the users whose data is sought, these processes offer limited guarantees that requests will be narrowly tailored or restricted to situations of real need. Law enforcement officials have argued that existing law allows access to even highly sensitive location information with these lesser safeguards, and the companies holding this information have generally complied. An ACLU records request of 250 police departments nationwide found that “virtually all” respondents said they track cell phone location data maintained by cellular companies, and “only a tiny minority reported consistently obtaining a warrant and demonstrating probable cause to do so.”

We believe this type of data access is unconstitutional. The Fourth Amendment requires the police to get a warrant before searching private places or documents, except in very limited emergency scenarios. The warrant requirement ensures that officials show “probable cause” for a search before an independent judge and specifically describe the information they’re seeking.

In 2012, the Supreme Court held that police trigger the Fourth Amendment’s protections when they install a GPS tracking device on a suspect’s car. Since then, law enforcement officials have argued that location data held by third parties is different. But we strongly believe that location data should be private regardless of where it’s stored, just as the Sixth Circuit has said for email, and we’re willing to go to court for that principle. Several courts are already considering this issue, and we look forward to the resolution of those cases.

We absolutely support and honor the hard work of law enforcement officials to protect citizens' safety and security. There’s no contradiction between this respect and ensuring that warrant processes are followed before our users' privacy is ever compromised. We hope other companies that hold location data will follow suit.

Thiago Santos joins Mapbox to help build the fastest in-device mapping library for the web and mobile. Thiago is joining us from Intel where he tuned major browsers for the Intel platform. Thiago’s drive is his huge passion for maps - for example he opened Brazil’s IBGE PDF maps for digitizing in OpenStreetMap.

Follow Thiago on GitHub and Twitter. Welcome Thiago!

Say hello to César Alfaro! César is joining the Mapbox team as a data specialist with a focus on improving OpenStreetMap based on customer feedback and automated error monitoring. Before joining Mapbox, César worked as database architect in the IT office of San Cristóbal of Huamanga University.

Follow César on GitHub and Twitter and see his work on OpenStreetMap.

Welcome César!

The Centre for Infrastructure, Sustainable Transportation and Urban Planning (CiSTUP) at the Indian Institute of Science is running a five day workshop on FOSS4G for urban environment applications starting today.

We will be talking about how to leverage OpenStreetMap followed by a hands-on session on Overpass, Mapbox Studio, Mapbox.js and Turf.js - full circle on collection, design and analysis of spatial data.

Our team in Bangalore is heads down working on improving open data sets like OpenStreetMap and we are looking for more people to join us.

Photo by Sanyam Bahga.