We provide a finite sample inference method for the structural parameters of the semiparametric binary response model under a conditional median restriction originally studied by Manski (1975, 1985). This is achieved by exploiting distributional properties of observable outcomes conditional on the observed sequence of exogenous variables. Moment inequalities conditional on the size n sequence of exogenous covariates are constructed, and the proposed test statistic is a monotone function of violations of the corresponding sample moment inequalities. The critical value used for inference is provided by the appropriate quantile of a known function of n independent Bernoulli random variables, and does not require the use of a cube root asymptotic approximation employing a point estimator of the target parameter. Simulation studies demonstrate favorable finite sample performance of the test in comparison to several existing approaches. Empirical use is illustrated with an application to the classical setting of transportation choice.