I'll address this from the medical literature perspective, I'm sure those in the more basic/bench sciences will have their own tidbits to add.

Just because something was done 'with the scientific method' does not mean that the results are 'true'. Even with a 'perfectly designed' study, biological systems are inherently extremely complex so we end up using p-values of 0.05 commonly, such that the chance of a falsely rejected null hypothesis is still substantial (especially in light of the sheer volume of studies published). Now add in limitations of study design and there are often dozens of individual issues that can weaken a paper's conclusion.

Let's say we're looking at a randomized control trial trying to demonstrate whether drug A is better than drug B at treating disease X. Even before looking at the results, we would have to analyze their inclusion/exclusion criteria (what kinds of patients are eligible for this study?), the control/intervention arms (do the interventions make sense?), the outcomes measured (are they reasonable and/or clinically valuable measures? is the length of follow-up appropriate?), the statistical analysis (is the study adequately powered? is the statistical analysis appropriate for this type of data?), and procedural steps (how was randomization done? was the source of funding appropriately separated from the design, data acquisition, and analysis?).

So now we're past the methods section of the paper and on to the results. The first figure usually describes the patient flow within the study. We look at it to see how patients progressed through the study, whether dropout rates could skew the study, and identify any other issues with the design. The next table typically shows patient demographics and demonstrates the results of randomization. We check to see whether the study population is applicable to our clinical question, and whether randomization was successful. If there are major differences between the groups then confounding variables may be involved. We can look at the individual figures showing the primary and secondary outcomes and ask whether the way they are presented is reasonable and in-line with what the data show. Finally, when we reach the conclusions section we can disagree with how the authors interpret their results and its generalizability towards our patient practice.

Clearly this is a huge topic, with different specific issues with different types of papers (eg prognostic study vs treatment study vs risk factor study etc) and different study designs (RCT vs cohort vs case control etc), but hopefully this gives an idea for the complexity of 1) designing a good study and 2) successfully writing it up. Even if everything above was done with the best of intentions limitations such as funding, disease prevalence, patient availability, logistical resources, etc can interfere and make a study less believable. Feel free to ask any further questions and I'll try to answer them below.