Very simply, the less of a copyrighted work you use, the more likely it is that your use will be considered fair. However, the court still considers the other factors, even if the amount of the work you use is very small. Even minor excerpts might still be infringing if it is considered to be the "heart" of the work. Likewise, some contexts might allow a much larger amount of the work to be used, such as a parody. This is why it is important to consider all four factors as a whole.
Click here to learn if you are protected by Fair Use